PetscSFConcatenate#

concatenate multiple PetscSF into one

Synopsis#

#include "petscsf.h" 
PetscErrorCode PetscSFConcatenate(MPI_Comm comm, PetscInt nsfs, PetscSF sfs[], PetscSFConcatenateRootMode rootMode, PetscInt leafOffsets[], PetscSF *newsf)

Input Parameters#

  • comm - the communicator

  • nsfs - the number of input PetscSF

  • sfs - the array of input PetscSF

  • rootMode - the root mode specifying how roots are handled

  • leafOffsets - the array of local leaf offsets, one for each input PetscSF, or NULL for contiguous storage

Output Parameters#

Notes#

The communicator of all SFs in sfs must be comm.

Leaves are always concatenated locally, keeping them ordered by the input SF index and original local order. The offsets in leafOffsets are added to the original leaf indices. If all input SFs use contiguous leaf storage (ilocal = NULL), leafOffsets can be passed as NULL as well. In this case, NULL is also passed as ilocal to the resulting SF. If any input SF has non-null ilocal, leafOffsets is needed to distinguish leaves from different input SFs. In this case, user is responsible to provide correct offsets so that the resulting leaves are unique (otherwise an error occurs).

All root modes retain the essential connectivity condition#

If two leaves of the same input SF are connected (sharing the same root), they are also connected in the output SF. Parameter rootMode controls how the input root spaces are combined. For PETSCSF_CONCATENATE_ROOTMODE_SHARED, the root space is considered the same for each input SF (checked in debug mode) and is also the same in the output SF. For PETSCSF_CONCATENATE_ROOTMODE_LOCAL and PETSCSF_CONCATENATE_ROOTMODE_GLOBAL, the input root spaces are taken as separate and joined. PETSCSF_CONCATENATE_ROOTMODE_LOCAL joins the root spaces locally; roots of sfs[0], sfs[1], sfs[2], … are joined on each rank separately, ordered by input SF and original local index, and renumbered contiguously. PETSCSF_CONCATENATE_ROOTMODE_GLOBAL joins the root spaces globally; roots of sfs[0], sfs[1], sfs[2, … are joined globally, ordered by input SF index and original global index, and renumbered contiguously; the original root ranks are ignored. For both PETSCSF_CONCATENATE_ROOTMODE_LOCAL and PETSCSF_CONCATENATE_ROOTMODE_GLOBAL, the output SF’s root layout is such that the local number of roots is a sum of the input SF’s local numbers of roots on each rank to keep the load balancing. However, for PETSCSF_CONCATENATE_ROOTMODE_GLOBAL, that means roots can move to different ranks.

Example#

We can use src/vec/is/sf/tests/ex18.c to compare the root modes. By running

make -C $PETSC_DIR/src/vec/is/sf/tests ex18
for m in {local,global,shared}; do
mpirun -n 2 $PETSC_DIR/src/vec/is/sf/tests/ex18 -nsfs 2 -n 2 -root_mode $m -sf_view
done

we generate two identical SFs sf_0 and sf_1,

PetscSF Object: sf_0 2 MPI processes
type: basic
rank #leaves #roots
[ 0]       4      2
[ 1]       4      2
leaves      roots       roots in global numbering
( 0,  0) <- ( 0,  0)  =   0
( 0,  1) <- ( 0,  1)  =   1
( 0,  2) <- ( 1,  0)  =   2
( 0,  3) <- ( 1,  1)  =   3
( 1,  0) <- ( 0,  0)  =   0
( 1,  1) <- ( 0,  1)  =   1
( 1,  2) <- ( 1,  0)  =   2
( 1,  3) <- ( 1,  1)  =   3

and pass them to PetscSFConcatenate() along with different choices of rootMode, yielding different result_sf#

rootMode = local:
PetscSF Object: result_sf 2 MPI processes
type: basic
rank #leaves #roots
[ 0]       8      4
[ 1]       8      4
leaves      roots       roots in global numbering
( 0,  0) <- ( 0,  0)  =   0
( 0,  1) <- ( 0,  1)  =   1
( 0,  2) <- ( 1,  0)  =   4
( 0,  3) <- ( 1,  1)  =   5
( 0,  4) <- ( 0,  2)  =   2
( 0,  5) <- ( 0,  3)  =   3
( 0,  6) <- ( 1,  2)  =   6
( 0,  7) <- ( 1,  3)  =   7
( 1,  0) <- ( 0,  0)  =   0
( 1,  1) <- ( 0,  1)  =   1
( 1,  2) <- ( 1,  0)  =   4
( 1,  3) <- ( 1,  1)  =   5
( 1,  4) <- ( 0,  2)  =   2
( 1,  5) <- ( 0,  3)  =   3
( 1,  6) <- ( 1,  2)  =   6
( 1,  7) <- ( 1,  3)  =   7

rootMode = global:
PetscSF Object: result_sf 2 MPI processes
type: basic
rank #leaves #roots
[ 0]       8      4
[ 1]       8      4
leaves      roots       roots in global numbering
( 0,  0) <- ( 0,  0)  =   0
( 0,  1) <- ( 0,  1)  =   1
( 0,  2) <- ( 0,  2)  =   2
( 0,  3) <- ( 0,  3)  =   3
( 0,  4) <- ( 1,  0)  =   4
( 0,  5) <- ( 1,  1)  =   5
( 0,  6) <- ( 1,  2)  =   6
( 0,  7) <- ( 1,  3)  =   7
( 1,  0) <- ( 0,  0)  =   0
( 1,  1) <- ( 0,  1)  =   1
( 1,  2) <- ( 0,  2)  =   2
( 1,  3) <- ( 0,  3)  =   3
( 1,  4) <- ( 1,  0)  =   4
( 1,  5) <- ( 1,  1)  =   5
( 1,  6) <- ( 1,  2)  =   6
( 1,  7) <- ( 1,  3)  =   7

rootMode = shared:
PetscSF Object: result_sf 2 MPI processes
type: basic
rank #leaves #roots
[ 0]       8      2
[ 1]       8      2
leaves      roots       roots in global numbering
( 0,  0) <- ( 0,  0)  =   0
( 0,  1) <- ( 0,  1)  =   1
( 0,  2) <- ( 1,  0)  =   2
( 0,  3) <- ( 1,  1)  =   3
( 0,  4) <- ( 0,  0)  =   0
( 0,  5) <- ( 0,  1)  =   1
( 0,  6) <- ( 1,  0)  =   2
( 0,  7) <- ( 1,  1)  =   3
( 1,  0) <- ( 0,  0)  =   0
( 1,  1) <- ( 0,  1)  =   1
( 1,  2) <- ( 1,  0)  =   2
( 1,  3) <- ( 1,  1)  =   3
( 1,  4) <- ( 0,  0)  =   0
( 1,  5) <- ( 0,  1)  =   1
( 1,  6) <- ( 1,  0)  =   2
( 1,  7) <- ( 1,  1)  =   3

See Also#

PetscSF, PetscSFCompose(), PetscSFGetGraph(), PetscSFSetGraph(), PetscSFConcatenateRootMode

Level#

advanced

Location#

src/vec/is/sf/interface/sf.c


Edit on GitLab

Index of all PetscSF routines
Table of Contents for all manual pages
Index of all manual pages