The initial process regions are non-overlapping and node locations are stored in Mesh_class::Mesh::eptnmap (element partition map).
The data needed for both cases and the distribution are done through the SpMtx_arrangement::SpMtx_DistributeAssembled subroutine. The preliminary distribution of data is done in SpMtx_arrangement::SpMtx_distributeWithOverlap and the exact data to be exchanged between processes is calculated in SpMtx_arrangement::SpMtx_build_ghost. We need to do two types of operations:
There are two cases which differ in their approach: non-overlapping and overlapping regions, so that even parallel matrix-vector multiplication is done by different subroutines.
With no overlap each mesh node belongs only to one process region. So called interface (ghost values) for region 1 is shown on the picture below.
Overlap is constructed by adding connected nodes in the mesh graph one layer at a time. The overlap of 2 (ol=2) means there will be 4 layers on the boundary that are shared by both processes. However, the ghost values are now in the last layer inside the region, not outside, so with ol=1 the ghost values are the same as in non-overlapping case. This is a trick which allows to reduce the amount of calculation in the overlapping case.
The general assumption is that the region values are up-to-date before and after each operation.
In non-overlapping case ghost values are sent before any calculations. In overlapping case, local values that are interface for any neighbour are computed first and sent out, only then the calculation of the remaining region is performed. This requires to distinguish the matrix values that are "ghost to local" in non-overlapping case and the matrix values that are "local to neighbour ghost" in overlapping case. The latter are shown on the following figure.
These are the values process 2 need to send to process 1 during matrix vector multiplication. Analogous values on process 1 for process 2 can be seen as symmetrical in respect to boundary.
Because the two cases are quite different and the optimizations the data structures are slightly complicated. Matrix values are divided into 6 categories:
The first 4 are marked by the the members
mtx_bbe of the SpMtx_class::SpMtx class. The 5. one is everything between
nnz. The last is stored to a separate matrix object
The vector indices that need to be sent and received during matrix-vector multiplication are stored in
ax_recvidx members of the Mesh_class::Mesh class. The vector indices that need to be exchanged (sent/received) during first level preconditioner are stored in