This is old DOUG wiki, please go to http://www.dougdevel.org/trac/wiki

From DOUGWiki

Here is a general description of doug_main behaviour from communication point of view. Pictures are screenshots of Jumpshot-4 performance visualiser (take a look at Profiling).

Picture 1. Overview of doug_main communication.

Picture 2. Zoom of phases 3 and 4 of doug_main communication and the Legend Map.

• During the 1st phase the master-process has read in the data from the input files and is sending the information about the task to slave processes. Sending the data takes time, so during the first fase slave processes wait (blocking MPI_Recv) until the information is received.

• The 2nd phase starts for the master-process as soon as the input information has been sent for the slaves (non-blocking MPI_ISend). It is happening around the 1st second in timeline (picture 1). Master is starting to build the dual-graph and once it is finished, master uses METIS to partition the dual graph. After that the master sends (non-blocking MPI_ISend) the information about the division of elements between the subdomains to slave processes (around second 9 in timeline). The second phase for the slave prosesses starts right after they have received the information they waited in the first phase (aroud second 12 in timeline). The slaves receive the information about the subdomains (around second 20 in timeline). At the end of the second phase each process does the second level local partitioning of the given subdomain and also detects its neighbours in a partitioned dual-graph (detects the neighbour subdomains).

• The 3rd phase is again starting earlier for master-process (around second 11 in timeline) as for the slave processes. During the third phase a local system matrix is being assembled. The master is sending (non-bloking MPI_ISend) the neccessary information to slaves, the slaves check if the information has been received (MPI_Probe) and then receive the information (MPI_Recv). On upper right corner of picture 2, there is a zoom of what happening from communication point of view during the third phase.

• • The phases 1-3 can be commonly called also as a "setup-phase".

• The 4th phase is the iterations' phase. The zoom of iterations' phase is on bottom right corner of the picture 2. During phase 4 there is non-blocking communication between the neighbours and also collective bloking communication within the whole communicator.

• The phase 5 is the finish-phase. The master process is receiving the information from slaves and sums up the result.