DOUG 0.2

The initial process regions are nonoverlapping and node locations are stored in Mesh_class::Mesh::eptnmap (element partition map).
The data needed for both cases and the distribution are done through the SpMtx_arrangement::SpMtx_DistributeAssembled subroutine. The preliminary distribution of data is done in SpMtx_arrangement::SpMtx_distributeWithOverlap and the exact data to be exchanged between processes is calculated in SpMtx_arrangement::SpMtx_build_ghost. We need to do two types of operations:
There are two cases which differ in their approach: nonoverlapping and overlapping regions, so that even parallel matrixvector multiplication is done by different subroutines.
With no overlap each mesh node belongs only to one process region. So called interface (ghost values) for region 1 is shown on the picture below.
Overlap is constructed by adding connected nodes in the mesh graph one layer at a time. The overlap of 2 (ol=2) means there will be 4 layers on the boundary that are shared by both processes. However, the ghost values are now in the last layer inside the region, not outside, so with ol=1 the ghost values are the same as in nonoverlapping case. This is a trick which allows to reduce the amount of calculation in the overlapping case.
The general assumption is that the region values are uptodate before and after each operation.
In nonoverlapping case ghost values are sent before any calculations. In overlapping case, local values that are interface for any neighbour are computed first and sent out, only then the calculation of the remaining region is performed. This requires to distinguish the matrix values that are "ghost to local" in nonoverlapping case and the matrix values that are "local to neighbour ghost" in overlapping case. The latter are shown on the following figure.
These are the values process 2 need to send to process 1 during matrix vector multiplication. Analogous values on process 1 for process 2 can be seen as symmetrical in respect to boundary.
Because the two cases are quite different and the optimizations the data structures are slightly complicated. Matrix values are divided into 6 categories:
The first 4 are marked by the the members mtx_bbs
and mtx_bbe
of the SpMtx_class::SpMtx class. The 5. one is everything between mtx_bbe(2,2)
and nnz
. The last is stored to a separate matrix object A_ghost
.
The vector indices that need to be sent and received during matrixvector multiplication are stored in ax_sendidx
and ax_recvidx
members of the Mesh_class::Mesh class. The vector indices that need to be exchanged (sent/received) during first level preconditioner are stored in ol_solve
array.