petsc-3.4.5 2014-06-29


Initialize additional processes to be used as "worker" processes. This is not generally called by users. One should use -hmpi_spawn_size <n> to indicate that you wish to have n-1 new MPI processes spawned for each current process.


#include "petscsys.h"   
PetscErrorCode  PetscHMPISpawn(PetscMPIInt nodesize)
Not Collective (could make collective on MPI_COMM_WORLD, generate one huge comm and then split it up)

Input Parameter

nodesize -size of each compute node that will share processors

Options Database

-hmpi_spawn_size nodesize - Notes: This is only supported on systems with an MPI 2 implementation that includes the MPI_Comm_Spawn() routine.

   Comparison of two approaches for HMPI usage (MPI started with N processes)

   -hmpi_spawn_size <n> requires MPI 2, results in n*N total processes with N directly used by application code
                                          and n-1 worker processes (used by PETSc) for each application node.
                          You MUST launch MPI so that only ONE MPI process is created for each hardware node.

   -hmpi_merge_size <n> results in N total processes, N/n used by the application code and the rest worker processes
                           (used by PETSc)
                          You MUST launch MPI so that n MPI processes are created for each hardware node.

   petscmpiexec -n 2 ./ex1 -hmpi_spawn_size 3 gives 2 application nodes (and 4 PETSc worker nodes)
   petscmpiexec -n 6 ./ex1 -hmpi_merge_size 3 gives the SAME 2 application nodes and 4 PETSc worker nodes
      This is what would use if each of the computers hardware nodes had 3 CPUs.

     These are intended to be used in conjunction with USER HMPI code. The user will have 1 process per
  computer (hardware) node (where the computer node has p cpus), the user's code will use threads to fully
  utilize all the CPUs on the node. The PETSc code will have p processes to fully use the compute node for
  PETSc calculations. The user THREADS and PETSc PROCESSES will NEVER run at the same time so the p CPUs
  are always working on p task, never more than p.

   See PCHMPI for a PETSc preconditioner that can use this functionality

For both PetscHMPISpawn() and PetscHMPIMerge() PETSC_COMM_WORLD consists of one process per "node", PETSC_COMM_LOCAL_WORLD consists of all the processes in a "node."

In both cases the user's code is running ONLY on PETSC_COMM_WORLD (that was newly generated by running this command).

See Also

PetscFinalize(), PetscInitializeFortran(), PetscGetArgs(), PetscHMPIFinalize(), PetscInitialize(), PetscHMPIMerge(), PetscHMPIRun()

Index of all Sys routines
Table of Contents for all manual pages
Index of all manual pages