Parallel Communication

Functions for parallel mesh operations and data transfer. More...

Collaboration diagram for Parallel Communication:

Functions
ErrCode	iMOAB_SendMesh (iMOAB_AppID pid, MPI_Comm *joint_communicator, MPI_Group *receivingGroup, int *rcompid, int *method)
	Transfer mesh data from sender component to receiver component. More...
ErrCode	iMOAB_ReceiveMesh (iMOAB_AppID pid, MPI_Comm *joint_communicator, MPI_Group *sendingGroup, int *scompid)
	Receive mesh data on target component from sending component. More...
ErrCode	iMOAB_SendElementTag (iMOAB_AppID pid, const iMOAB_String tag_storage_name, MPI_Comm *joint_communicator, int *context_id)
	Send element tag data from calling component to another component. More...
ErrCode	iMOAB_ReceiveElementTag (iMOAB_AppID pid, const iMOAB_String tag_storage_name, MPI_Comm *joint_communicator, int *context_id)
	Receive element tag data from another component. More...
ErrCode	iMOAB_ComputeCommGraph (iMOAB_AppID pid1, iMOAB_AppID pid2, MPI_Comm *joint_communicator, MPI_Group *group1, MPI_Group *group2, int *type1, int *type2, int *comp1, int *comp2)
	Compute communication graph between two components using rendezvous algorithm. More...
ErrCode	iMOAB_MergeVertices (iMOAB_AppID pid)
	Merge duplicate vertices in a mesh and update connectivity information. More...

Detailed Description

Functions for parallel mesh operations and data transfer.

Function Documentation

iMOAB_ComputeCommGraph()

ErrCode iMOAB_ComputeCommGraph	(	iMOAB_AppID	pid1,
		iMOAB_AppID	pid2,
		MPI_Comm *	joint_communicator,
		MPI_Group *	group1,
		MPI_Group *	group2,
		int *	type1,
		int *	type2,
		int *	comp1,
		int *	comp2
	)

Compute communication graph between two components using rendezvous algorithm.

Implements sophisticated rendezvous-based algorithm to determine which processes from component 1 need to communicate with which processes from component 2. Uses global entity IDs (or DOFs) as "rendezvous points" to discover communication patterns between components.

Algorithm Workflow:

1. COMMUNICATOR SETUP: Resolve MPI communicator and group handles (Fortran F2C conversions), create bidirectional ParCommGraph instances for both components, store graphs in pgraph maps.
2. ENTITY ID COLLECTION: Extract entity IDs based on type parameter:
   • type=1: GLOBAL_DOFS from MBQUAD elements (spectral elements)
   • type=2: GLOBAL_ID from MBVERTEX entities (vertex-based coupling)
   • type=3: GLOBAL_ID from 2D elements (finite volume meshes)
   • Handle TempestRemap coverage mesh entities
3. RENDEZVOUS ALGORITHM: Create TupleList for each component with (target_proc, entity_id) pairs. Use hash-based distribution (entity_id % numProcs). Execute crystal router (gs_transfer) to send entity IDs to rendezvous processes. Sort received data by entity ID for efficient matching.
4. COMMUNICATION DISCOVERY: Synchronously iterate through both sorted TupleLists, find matching entity IDs between components (intersection), record which processes share each entity, build reverse communication maps.
5. GRAPH CONSTRUCTION: Send communication information back to original processes via crystal router. Each process learns communication partners. Store patterns in ParCommGraph.

Data Processed:

• Entity IDs: Global identifiers serving as rendezvous points
• Process mappings: Which processes own which entities
• Communication patterns: Sender→receiver relationships

Parameters

[in]	pid1	Application ID for component 1 (or negative if not on these ranks)
[in]	pid2	Application ID for component 2 (or negative if not on these ranks)
[in]	joint_communicator	MPI communicator spanning both component groups
[in]	group1	MPI group for component 1 processes
[in]	group2	MPI group for component 2 processes
[in]	type1	Entity type for component 1 (1=DOFs, 2=vertices, 3=elements)
[in]	type2	Entity type for component 2 (1=DOFs, 2=vertices, 3=elements)
[in]	comp1	Component 1 external ID
[in]	comp2	Component 2 external ID

Note

Complexity: O(N log N) sorting, O(N/P) communication per process

Use cases: Coupling different mesh types, establishing comm for data transfer, mesh intersection prep

Warning

Both components must have consistent global entity IDs for rendezvous to work

Returns

moab::MB_SUCCESS on success, error code otherwise

Definition at line 3753 of file iMOAB.cpp.

{
    // Validate input parameters
    assert( joint_communicator );
    assert( group1 );
    assert( group2 );
    int localRank = 0, numProcs = 1;
 
    // Determine if either component is using Fortran (affects MPI handle conversion)
    bool isFortran = false;
    if( *pid1 >= 0 ) isFortran = isFortran || context.appDatas[*pid1].is_fortran;
    if( *pid2 >= 0 ) isFortran = isFortran || context.appDatas[*pid2].is_fortran;
 
    // Handle Fortran-to-C MPI handle conversion if needed
    MPI_Comm global =
        ( isFortran ? MPI_Comm_f2c( *reinterpret_cast< MPI_Fint* >( joint_communicator ) ) : *joint_communicator );
    MPI_Group srcGroup = ( isFortran ? MPI_Group_f2c( *reinterpret_cast< MPI_Fint* >( group1 ) ) : *group1 );
    MPI_Group tgtGroup = ( isFortran ? MPI_Group_f2c( *reinterpret_cast< MPI_Fint* >( group2 ) ) : *group2 );
 
    // Get process information for the joint communicator
    MPI_Comm_rank( global, &localRank );
    MPI_Comm_size( global, &numProcs );
 
    // Create bidirectional communication graphs for both components
 
    // Create communication graphs for both components (bidirectional)
    ParCommGraph* cgraph     = nullptr;
    ParCommGraph* cgraph_rev = nullptr;
 
    if( *pid1 >= 0 )
    {
        // Create communication graph for component 1 -> component 2
        appData& data = context.appDatas[*pid1];
        auto mt       = data.pgraph.find( *comp2 );
        if( mt != data.pgraph.end() ) data.pgraph.erase( mt );  // Remove existing graph if present
        cgraph                                 = new ParCommGraph( global, srcGroup, tgtGroup, *comp1, *comp2 );
        context.appDatas[*pid1].pgraph[*comp2] = cgraph;  // Store with target component ID as key
    }
 
    if( *pid2 >= 0 )
    {
        // Create reverse communication graph for component 2 -> component 1
        appData& data = context.appDatas[*pid2];
        auto mt       = data.pgraph.find( *comp1 );
        if( mt != data.pgraph.end() ) data.pgraph.erase( mt );  // Remove existing graph if present
        cgraph_rev                             = new ParCommGraph( global, tgtGroup, srcGroup, *comp2, *comp1 );
        context.appDatas[*pid2].pgraph[*comp1] = cgraph_rev;  // Store with source component ID as key
    }
 
    // Initialize TupleLists for rendezvous algorithm
    // Each tuple contains: (target_process, entity_id) pairs for hash-based distribution
    TupleList TLcomp1;
    TLcomp1.initialize( 2, 0, 0, 0, 0 );  // 2 integers: target_proc, entity_id
    TupleList TLcomp2;
    TLcomp2.initialize( 2, 0, 0, 0, 0 );  // 2 integers: target_proc, entity_id
 
    // Enable write access for building the tuple lists
    TLcomp1.enableWriteAccess();
 
    // Get tag handles for entity identification
    // GLOBAL_DOFS: for spectral elements (type 1)
    // GLOBAL_ID: for vertices (type 2) and 2D elements (type 3)
    Tag gdsTag;
    int lenTagType1 = 1;
    if( 1 == *type1 || 1 == *type2 )
    {
        // Get GLOBAL_DOFS tag for spectral elements (usually 16 DOFs per element)
        MB_CHK_ERR( context.MBI->tag_get_handle( "GLOBAL_DOFS", gdsTag ) );
        MB_CHK_ERR( context.MBI->tag_get_length( gdsTag, lenTagType1 ) );  // Usually 16 DOFs per element
    }
    Tag gidTag = context.MBI->globalId_tag();  // Standard GLOBAL_ID tag
 
    // Collect entity IDs from component 1 for rendezvous algorithm
    std::vector< int > valuesComp1;
    if( *pid1 >= 0 )
    {
        appData& data1     = context.appDatas[*pid1];
        EntityHandle fset1 = data1.file_set;
 
        // Handle TempestRemap coverage mesh for intersection applications
#ifdef MOAB_HAVE_TEMPESTREMAP
        if( data1.tempestData.remapper != nullptr )  // This is the case for intersection operations
            fset1 = data1.tempestData.remapper->GetMeshSet( Remapper::CoveringMesh );
#endif
 
        Range ents_of_interest;
        if( *type1 == 1 )  // Spectral elements: get GLOBAL_DOFS from MBQUAD elements
        {
            assert( gdsTag );
            MB_CHK_ERR( context.MBI->get_entities_by_type( fset1, MBQUAD, ents_of_interest ) );
            valuesComp1.resize( ents_of_interest.size() * lenTagType1 );
            MB_CHK_ERR( context.MBI->tag_get_data( gdsTag, ents_of_interest, &valuesComp1[0] ) );
        }
        else if( *type1 == 2 )  // Vertex-based coupling: get GLOBAL_ID from MBVERTEX entities
        {
            MB_CHK_ERR( context.MBI->get_entities_by_type( fset1, MBVERTEX, ents_of_interest ) );
            valuesComp1.resize( ents_of_interest.size() );
            MB_CHK_ERR( context.MBI->tag_get_data( gidTag, ents_of_interest, &valuesComp1[0] ) );
        }
        else if( *type1 == 3 )  // Finite volume meshes: get GLOBAL_ID from 2D elements
        {
            MB_CHK_ERR( context.MBI->get_entities_by_dimension( fset1, 2, ents_of_interest ) );
            valuesComp1.resize( ents_of_interest.size() );
            MB_CHK_ERR( context.MBI->tag_get_data( gidTag, ents_of_interest, &valuesComp1[0] ) );
        }
        else
        {
            MB_CHK_ERR( MB_FAILURE );  // Only types 1, 2, or 3 are supported
        }
 
        // Build tuple list for component 1: (target_process, entity_id) pairs
        // Use hash-based distribution: entity_id % numProcs determines target process
        std::set< int > uniq( valuesComp1.begin(), valuesComp1.end() );  // Remove duplicates
        TLcomp1.resize( uniq.size() );
        for( std::set< int >::iterator sit = uniq.begin(); sit != uniq.end(); sit++ )
        {
            int marker               = *sit;               // Entity ID
            int to_proc              = marker % numProcs;  // Hash-based target process
            int n                    = TLcomp1.get_n();
            TLcomp1.vi_wr[2 * n]     = to_proc;  // Target process
            TLcomp1.vi_wr[2 * n + 1] = marker;   // Entity ID
            TLcomp1.inc_n();
        }
    }
 
    // Execute crystal router transfer for component 1 entity IDs
    // This sends entity IDs to rendezvous processes based on hash distribution
    ProcConfig pc( global );                            // Process configuration for crystal router
    pc.crystal_router()->gs_transfer( 1, TLcomp1, 0 );  // Transfer to joint tasks with markers
 
    // Sort component 1 tuple list by entity ID (key 1) for efficient matching
#ifdef VERBOSE
    std::stringstream ff1;
    ff1 << "TLcomp1_" << localRank << ".txt";
    TLcomp1.print_to_file( ff1.str().c_str() );  // Debug output before sorting
#endif
    moab::TupleList::buffer sort_buffer;
    sort_buffer.buffer_init( TLcomp1.get_n() );
    TLcomp1.sort( 1, &sort_buffer );  // Sort by entity ID (column 1)
    sort_buffer.reset();
#ifdef VERBOSE
    TLcomp1.print_to_file( ff1.str().c_str() );  // Debug output after sorting
#endif
    // Now process component 2 in the same way as component 1
    TLcomp2.enableWriteAccess();
 
    // Collect entity IDs from component 2 for rendezvous algorithm
    std::vector< int > valuesComp2;
    if( *pid2 >= 0 )
    {
        appData& data2     = context.appDatas[*pid2];
        EntityHandle fset2 = data2.file_set;
 
        // Handle TempestRemap coverage mesh for intersection applications
#ifdef MOAB_HAVE_TEMPESTREMAP
        if( data2.tempestData.remapper != nullptr )  // This is the case for intersection operations
            fset2 = data2.tempestData.remapper->GetMeshSet( Remapper::CoveringMesh );
#endif
 
        Range ents_of_interest;
        if( *type2 == 1 )  // Spectral elements: get GLOBAL_DOFS from MBQUAD elements
        {
            assert( gdsTag );
            MB_CHK_ERR( context.MBI->get_entities_by_type( fset2, MBQUAD, ents_of_interest ) );
            valuesComp2.resize( ents_of_interest.size() * lenTagType1 );
            MB_CHK_ERR( context.MBI->tag_get_data( gdsTag, ents_of_interest, &valuesComp2[0] ) );
        }
        else if( *type2 == 2 )  // Vertex-based coupling: get GLOBAL_ID from MBVERTEX entities
        {
            MB_CHK_ERR( context.MBI->get_entities_by_type( fset2, MBVERTEX, ents_of_interest ) );
            valuesComp2.resize( ents_of_interest.size() );
            MB_CHK_ERR( context.MBI->tag_get_data( gidTag, ents_of_interest, &valuesComp2[0] ) );
        }
        else if( *type2 == 3 )  // Finite volume meshes: get GLOBAL_ID from 2D elements
        {
            MB_CHK_ERR( context.MBI->get_entities_by_dimension( fset2, 2, ents_of_interest ) );
            valuesComp2.resize( ents_of_interest.size() );
            MB_CHK_ERR( context.MBI->tag_get_data( gidTag, ents_of_interest, &valuesComp2[0] ) );
        }
        else
        {
            MB_CHK_ERR( MB_FAILURE );  // Only types 1, 2, or 3 are supported
        }
        // Build tuple list for component 2: (target_process, entity_id) pairs
        // Use hash-based distribution: entity_id % numProcs determines target process
        std::set< int > uniq( valuesComp2.begin(), valuesComp2.end() );  // Remove duplicates
        TLcomp2.resize( uniq.size() );
        for( std::set< int >::iterator sit = uniq.begin(); sit != uniq.end(); sit++ )
        {
            int marker               = *sit;               // Entity ID
            int to_proc              = marker % numProcs;  // Hash-based target process
            int n                    = TLcomp2.get_n();
            TLcomp2.vi_wr[2 * n]     = to_proc;  // Target process
            TLcomp2.vi_wr[2 * n + 1] = marker;   // Entity ID
            TLcomp2.inc_n();
        }
    }
 
    // Execute crystal router transfer for component 2 entity IDs
    pc.crystal_router()->gs_transfer( 1, TLcomp2, 0 );  // Transfer to joint tasks with markers
 
    // Sort component 2 tuple list by entity ID (key 1) for efficient matching
#ifdef VERBOSE
    std::stringstream ff2;
    ff2 << "TLcomp2_" << localRank << ".txt";
    TLcomp2.print_to_file( ff2.str().c_str() );  // Debug output before sorting
#endif
    sort_buffer.buffer_reserve( TLcomp2.get_n() );
    TLcomp2.sort( 1, &sort_buffer );  // Sort by entity ID (column 1)
    sort_buffer.reset();
#ifdef VERBOSE
    TLcomp2.print_to_file( ff2.str().c_str() );  // Debug output after sorting
#endif
    // RENDEZVOUS ALGORITHM: Find matching entity IDs between components
    // Now we need to send back communication information from the rendezvous point
    // Loop synchronously over both sorted tuple lists to find matching entity IDs
    // Build new tuple lists to send communication information back to original processes
 
    // Tuple lists for sending communication info back to components
    // Format: (target_process, entity_id, source_process_from_other_component)
    TupleList TLBackToComp1;
    TLBackToComp1.initialize( 3, 0, 0, 0, 0 );  // 3 integers: target_proc, entity_id, source_proc_from_comp2
    TLBackToComp1.enableWriteAccess();
 
    TupleList TLBackToComp2;
    TLBackToComp2.initialize( 3, 0, 0, 0, 0 );  // 3 integers: target_proc, entity_id, source_proc_from_comp1
    TLBackToComp2.enableWriteAccess();
 
    // Get sizes of both tuple lists for synchronized iteration
    int n1 = TLcomp1.get_n();
    int n2 = TLcomp2.get_n();
 
    // Synchronized iteration through both sorted tuple lists to find matching entity IDs
    int indexInTLComp1 = 0;
    int indexInTLComp2 = 0;
    if( n1 > 0 && n2 > 0 )
    {
        while( indexInTLComp1 < n1 && indexInTLComp2 < n2 )  // Continue until either list is exhausted
        {
            // Get current entity IDs from both components
            int currentValue1 = TLcomp1.vi_rd[2 * indexInTLComp1 + 1];  // Entity ID from comp1
            int currentValue2 = TLcomp2.vi_rd[2 * indexInTLComp2 + 1];  // Entity ID from comp2
 
            if( currentValue1 < currentValue2 )
            {
                // Entity ID exists in comp1 but not in comp2 - skip comp1 entry
                // This is normal for non-overlapping meshes
                indexInTLComp1++;
                continue;
            }
            if( currentValue1 > currentValue2 )
            {
                // Entity ID exists in comp2 but not in comp1 - skip comp2 entry
                // This is normal for non-overlapping meshes
                indexInTLComp2++;
                continue;
            }
            // Found matching entity ID! Count consecutive entries with same ID in both lists
            int size1 = 1;
            int size2 = 1;
            while( indexInTLComp1 + size1 < n1 && currentValue1 == TLcomp1.vi_rd[2 * ( indexInTLComp1 + size1 ) + 1] )
                size1++;  // Count consecutive entries with same entity ID in comp1
            while( indexInTLComp2 + size2 < n2 && currentValue2 == TLcomp2.vi_rd[2 * ( indexInTLComp2 + size2 ) + 1] )
                size2++;  // Count consecutive entries with same entity ID in comp2
 
            // Create communication pairs for all combinations of processes that share this entity ID
            for( int i1 = 0; i1 < size1; i1++ )
            {
                for( int i2 = 0; i2 < size2; i2++ )
                {
                    // Send communication info back to component 1 processes
                    int n = TLBackToComp1.get_n();
                    TLBackToComp1.reserve();
                    TLBackToComp1.vi_wr[3 * n] =
                        TLcomp1.vi_rd[2 * ( indexInTLComp1 + i1 )];  // Target process from comp1
                    TLBackToComp1.vi_wr[3 * n + 1] = currentValue1;  // Shared entity ID
                    TLBackToComp1.vi_wr[3 * n + 2] =
                        TLcomp2.vi_rd[2 * ( indexInTLComp2 + i2 )];  // Source process from comp2
 
                    // Send communication info back to component 2 processes
                    n = TLBackToComp2.get_n();
                    TLBackToComp2.reserve();
                    TLBackToComp2.vi_wr[3 * n] =
                        TLcomp2.vi_rd[2 * ( indexInTLComp2 + i2 )];  // Target process from comp2
                    TLBackToComp2.vi_wr[3 * n + 1] = currentValue1;  // Shared entity ID
                    TLBackToComp2.vi_wr[3 * n + 2] =
                        TLcomp1.vi_rd[2 * ( indexInTLComp1 + i1 )];  // Source process from comp1
                }
            }
 
            // Advance indices past all entries with the current entity ID
            indexInTLComp1 += size1;
            indexInTLComp2 += size2;
        }
    }
    // Send communication information back to original component processes
    pc.crystal_router()->gs_transfer( 1, TLBackToComp1, 0 );  // Send to component 1 processes
    pc.crystal_router()->gs_transfer( 1, TLBackToComp2, 0 );  // Send to component 2 processes
 
    // Process communication information on component 1 processes
    if( *pid1 >= 0 )
    {
        // Sort received communication information by entity ID for efficient processing
#ifdef VERBOSE
        std::stringstream f1;
        f1 << "TLBack1_" << localRank << ".txt";
        TLBackToComp1.print_to_file( f1.str().c_str() );  // Debug output before sorting
#endif
        sort_buffer.buffer_reserve( TLBackToComp1.get_n() );
        TLBackToComp1.sort( 1, &sort_buffer );  // Sort by entity ID (column 1)
        sort_buffer.reset();
#ifdef VERBOSE
        TLBackToComp1.print_to_file( f1.str().c_str() );  // Debug output after sorting
#endif
 
        // Set up communication graph for component 1 based on discovered communication patterns
        // This establishes which processes component 1 needs to communicate with
        cgraph->settle_comm_by_ids( *comp1, TLBackToComp1, valuesComp1 );
    }
    // Process communication information on component 2 processes
    if( *pid2 >= 0 )
    {
        // Sort received communication information by source process for efficient processing
#ifdef VERBOSE
        std::stringstream f2;
        f2 << "TLBack2_" << localRank << ".txt";
        TLBackToComp2.print_to_file( f2.str().c_str() );  // Debug output before sorting
#endif
        sort_buffer.buffer_reserve( TLBackToComp2.get_n() );
        TLBackToComp2.sort( 2, &sort_buffer );  // Sort by source process (column 2)
        sort_buffer.reset();
#ifdef VERBOSE
        TLBackToComp2.print_to_file( f2.str().c_str() );  // Debug output after sorting
#endif
 
        // Set up reverse communication graph for component 2 based on discovered communication patterns
        // This establishes which processes component 2 needs to communicate with
        cgraph_rev->settle_comm_by_ids( *comp2, TLBackToComp2, valuesComp2 );
    }
 
    // Communication graph computation complete - both components now know their communication partners
    return moab::MB_SUCCESS;
}

References GlobalContext::appDatas, context, moab::Remapper::CoveringMesh, moab::ProcConfig::crystal_router(), moab::TupleList::enableWriteAccess(), appData::file_set, moab::Interface::get_entities_by_dimension(), moab::Interface::get_entities_by_type(), moab::TupleList::get_n(), moab::Interface::globalId_tag(), moab::TupleList::inc_n(), moab::TupleList::initialize(), MB_CHK_ERR, MB_SUCCESS, GlobalContext::MBI, MBQUAD, MBVERTEX, appData::pgraph, moab::TupleList::print_to_file(), moab::TupleList::reserve(), moab::TupleList::buffer::reset(), moab::TupleList::resize(), moab::ParCommGraph::settle_comm_by_ids(), moab::Range::size(), moab::TupleList::sort(), moab::Interface::tag_get_data(), moab::Interface::tag_get_handle(), moab::Interface::tag_get_length(), moab::TupleList::vi_rd, and moab::TupleList::vi_wr.

iMOAB_MergeVertices()

ErrCode iMOAB_MergeVertices

(

iMOAB_AppID

pid

)

Merge duplicate vertices in a mesh and update connectivity information.

This function consolidates duplicate vertices that may exist after mesh operations such as intersection or migration. Vertices with identical GLOBAL_ID values are merged, and connectivity of elements is updated accordingly. The function performs both local (within-process) and parallel (across-process) vertex merging.

Algorithm Workflow:

1. TAG COLLECTION: Collect critical tags that need to be preserved during merge:
   • GLOBAL_ID: Required for vertex identification (fatal error if missing)
   • area: Optional - element area values to preserve
   • frac: Optional - fraction values for conservation
2. LOCAL VERTEX DEDUPLICATION: Remove duplicate vertices within each process using spatial tolerance (1.0e-9). IntxUtils::remove_duplicate_vertices merges vertices and updates connectivity, preserving tag values from first occurrence.
3. MATERIAL SET CLEANUP: Clear all elements from material sets and re-populate with valid elements from the current file set. Ensures material set integrity after topology changes.
4. MESH INFO UPDATE: Refresh internal mesh information (vertex counts, element counts, owned/ghosted entity information) and recalculate mesh statistics.
5. PARALLEL VERTEX MERGING: ParallelMergeMesh handles vertices shared across process boundaries. Identifies vertices with identical GLOBAL_ID on different processes and merges them while updating parallel communication graph.
6. GLOBAL ID ASSIGNMENT: Reassign consistent global IDs to vertices after merge. Only updates vertex global IDs (cells already have correct IDs).
7. PARTITION TAG SETUP: Create/retrieve PARALLEL_PARTITION tag for visualization indicating which process owns each mesh set.

Data Processed:

• Vertices: Spatial coordinates and associated tags
• Elements: Connectivity information referencing vertices
• Tags: GLOBAL_ID (required), area, frac (optional)
• Material sets: Collections of elements with shared properties

Common Use Cases:

• After mesh intersection: Overlapping meshes create duplicate vertices
• After mesh migration: Vertices copied to multiple processes need merging
• After remapping: Source/target vertices may coincide
• Quality improvement: Remove numerical duplicates from mesh operations

Parameters

[in]

pid

Application ID for the mesh

Precondition

GLOBAL_ID tag must exist on vertices

Postcondition

Duplicate vertices merged both locally and across processes

Element connectivity updated to reference merged vertices

PARALLEL_PARTITION tag set for visualization

Note

Parallelism: Performs both local (per-process) and parallel (across-process) merging

Warning

Modifies mesh topology and connectivity - may invalidate cached entity handles

Returns

moab::MB_SUCCESS on success, moab::MB_FAILURE if GLOBAL_ID tag missing

Definition at line 4158 of file iMOAB.cpp.

{
    // =========================================================================
    // Validate inputs and initialize local variables
    // =========================================================================
 
    // Get application data and parallel communicator for this component
    // data contains mesh info, ranges, and pcomm holds parallel communication state
    appData& data     = context.appDatas[*pid];
    ParallelComm* pco = data.pcomm;
 
    // Initialize return value to success state
    ErrorCode rval = MB_SUCCESS;
 
    // =========================================================================
    // STEP 1: Collect tags that need to be preserved during vertex merging
    // =========================================================================
 
    // Initialize vector to hold tags that should be preserved during merge
    // These tags contain important data that must be maintained when vertices are merged
    // The merge operation will preserve tag values from the first vertex in each duplicate set
    std::vector< Tag > tagsList;
    Tag tag;
 
    // GLOBAL_ID is mandatory - used to identify duplicate vertices across processes
    // Without this tag, we cannot determine which vertices should be merged
    tag = context.MBI->globalId_tag();
    if( !tag )
    {
        // Fatal error: GLOBAL_ID tag is required for merge operations
        // Return failure as merge cannot proceed without vertex identification capability
        return moab::MB_FAILURE;
    }
    // Add GLOBAL_ID to the list of tags to preserve during merge
    tagsList.push_back( tag );
 
    // Area tag (optional) - contains element area values for conservation checks
    // Used in remapping applications to ensure conservation properties
    rval = context.MBI->tag_get_handle( "area", tag );
    if( tag && MB_SUCCESS == rval )
    {
        // Area tag found, include it in preserved tags
        tagsList.push_back( tag );
    }
 
    // frac tag (optional) - contains fractional coverage for remapping conservation
    // Used to track fractional ownership in intersection-based remapping
    rval = context.MBI->tag_get_handle( "frac", tag );
    if( tag && MB_SUCCESS == rval )
    {
        // Fraction tag found, include it in preserved tags
        tagsList.push_back( tag );
    }
 
    // ========================================================================
    // STEP 2: Remove duplicate vertices within this process (local merge)
    // ========================================================================
 
    // Set spatial tolerance for considering vertices identical (meters on unit sphere)
    // This tolerance accounts for numerical precision issues in floating-point coordinates
    // 1.0e-9 is a typical tolerance for spherical meshes near unit radius
    const double tol = 1.0e-9;
 
    // Perform local vertex deduplication using MOAB's IntxUtils
    // This function:
    // 1. Identifies spatially coincident vertices within tolerance
    // 2. Updates element connectivity to reference merged vertices
    // 3. Preserves tag values from the first vertex in each duplicate set
    // 4. Removes duplicate vertices from the mesh
    MB_CHK_ERR( IntxUtils::remove_duplicate_vertices( context.MBI, data.file_set, tol,
                                                      tagsList ) );  // Exit on failure to merge local vertices
 
    // ========================================================================
    // STEP 3: Clean up material sets that may reference deleted elements
    // ========================================================================
 
    // Material sets organize elements by material properties (e.g., land, ocean, ice)
    // After vertex merge, some elements may have been deleted or connectivity modified
    // Update material sets to maintain mesh data integrity
 
    // Retrieve all material sets currently defined on the file set
    // Material sets are identified by having a MATERIAL_SET_TAG_NAME tag
    MB_CHK_ERR( context.MBI->get_entities_by_type_and_tag(
        data.file_set, MBENTITYSET, &( context.material_tag ), nullptr, 1, data.mat_sets,
        Interface::UNION ) );  // Exit if we can't retrieve material sets
 
    // Update material sets if any were found
    if( !data.mat_sets.empty() )
    {
        // For simplicity, we only clean the first material set
        // In production code, all material sets should be systematically updated
        // This is a limitation that could be improved in future versions
        EntityHandle matSet = data.mat_sets[0];
        Range elems;
 
        // Get current 2D elements in this material set (may include stale references)
        MB_CHK_ERR(
            context.MBI->get_entities_by_dimension( matSet, 2, elems ) );  // Exit if we can't get material set contents
 
        // Remove all current elements from the material set
        // This clears potentially stale element references
        MB_CHK_ERR( context.MBI->remove_entities( matSet, elems ) );  // Exit on failure to remove stale elements
 
        // Get only valid 2D elements from the current file set
        // This excludes any elements that may have been deleted during merge
        elems.clear();  // Reset range for reuse
        MB_CHK_ERR(
            context.MBI->get_entities_by_dimension( data.file_set, 2, elems ) );  // Exit if we can't get valid elements
 
        // Add back only the valid elements to the material set
        // This ensures material set consistency after topology changes
        MB_CHK_ERR( context.MBI->add_entities( matSet, elems ) );  // Exit on failure to update material set
    }
 
    // ========================================================================
    // STEP 4: Update internal mesh information after topology changes
    // ========================================================================
 
    // Recalculate mesh statistics and update internal data structures
    // This includes vertex/element counts, owned/ghosted entity information
    // iMOAB_UpdateMeshInfo() refreshes all cached mesh information
    MB_CHK_ERR( iMOAB_UpdateMeshInfo( pid ) );  // Exit on failure to update mesh info
 
    // ========================================================================
    // STEP 5: Perform parallel merge of vertices shared across processes
    // ========================================================================
 
    // Create parallel merge mesh object with the same spatial tolerance
    // ParallelMergeMesh handles vertices shared across process boundaries
    ParallelMergeMesh pmm( pco, tol );
 
    // Perform parallel vertex merging across all processes
    // This identifies vertices with identical GLOBAL_ID on different processes
    // and merges them while updating the parallel communication graph
    // Parameters:
    //   - data.file_set: The mesh set to operate on
    //   - false: Skip local merge (already done in step 2)
    //   - 2: Update 2D elements after merge (dimension of elements to fix connectivity for)
    MB_CHK_ERR( pmm.merge( data.file_set,
                           /* do not do local merge*/ false,
                           /*  2d cells*/ 2 ) );  // Exit on failure of parallel merge
 
    // ========================================================================
    // STEP 6: Assign consistent global IDs to vertices after merge
    // ========================================================================
 
    // After merging, vertices need new globally unique IDs to maintain consistency
    // Only update vertices (dimension 0) - elements already have correct IDs
    // assign_global_ids() coordinates across all processes to ensure uniqueness
    MB_CHK_ERR( pco->assign_global_ids( data.file_set, /*dim*/ 0 ) );  // Exit on failure to assign global IDs
 
    // ========================================================================
    // STEP 7: Set partition tag for visualization and debugging
    // ========================================================================
 
    // Set up PARALLEL_PARTITION tag to indicate which process owns each mesh set
    // This is essential for visualization tools like ParaView/VisIt to show
    // domain decomposition and parallel partitioning
 
    Tag part_tag;     // Handle for the partition tag
    int dum_id = -1;  // Default value if tag doesn't exist yet
 
    // Create or retrieve the partition tag (sparse storage, one int per set)
    // Sparse storage is efficient since we only need one value per mesh set
    rval = context.MBI->tag_get_handle( "PARALLEL_PARTITION", 1, MB_TYPE_INTEGER, part_tag,
                                        MB_TAG_CREAT | MB_TAG_SPARSE, &dum_id );
 
    // Validate that partition tag was created or retrieved successfully
    if( part_tag == nullptr || ( ( MB_SUCCESS != rval ) && ( MB_ALREADY_ALLOCATED != rval ) ) )
    {
        // Error creating/retrieving partition tag
        std::cout << "Failed to create/retrieve PARALLEL_PARTITION tag.\n";
        return moab::MB_FAILURE;
    }
 
    // Tag the file set with this process's rank to show ownership
    // This allows visualization tools to distinguish between different process domains
    int rank = pco->rank();
    MB_CHK_ERR(
        context.MBI->tag_set_data( part_tag, &data.file_set, 1, &rank ) );  // Exit on failure to set partition data
 
    // ========================================================================
    // FUNCTION EPILOGUE: Return success status
    // ========================================================================
 
    // All merge operations completed successfully
    return moab::MB_SUCCESS;
}

References moab::Interface::add_entities(), GlobalContext::appDatas, moab::ParallelComm::assign_global_ids(), moab::Range::clear(), context, moab::Range::empty(), ErrorCode, appData::file_set, moab::Interface::get_entities_by_dimension(), moab::Interface::get_entities_by_type_and_tag(), moab::Interface::globalId_tag(), iMOAB_UpdateMeshInfo(), appData::mat_sets, GlobalContext::material_tag, MB_ALREADY_ALLOCATED, MB_CHK_ERR, MB_SUCCESS, MB_TAG_CREAT, MB_TAG_SPARSE, MB_TYPE_INTEGER, MBENTITYSET, GlobalContext::MBI, moab::ParallelMergeMesh::merge(), appData::pcomm, moab::ParallelComm::rank(), moab::IntxUtils::remove_duplicate_vertices(), moab::Interface::remove_entities(), moab::Interface::tag_get_handle(), moab::Interface::tag_set_data(), and moab::Interface::UNION.

iMOAB_ReceiveElementTag()

ErrCode iMOAB_ReceiveElementTag	(	iMOAB_AppID	pid,
		const iMOAB_String	tag_storage_name,
		MPI_Comm *	joint_communicator,
		int *	context_id
	)

Receive element tag data from another component.

Receives tag data (scalar or vector values) associated with mesh elements from another component using pre-established communication graph. Received tag values are applied to corresponding entities in local mesh, enabling data exchange between coupled components.

Algorithm Workflow:

1. COMMUNICATION GRAPH VALIDATION: Look up ParCommGraph for specified context_id, validate existence and initialization, extract graph and parallel communicator.
2. ENTITY SELECTION: Determine target entities based on component type (point cloud: local_verts, regular mesh: owned_elems). Handle special cases (coverage set, TempestRemap coverage mesh).
3. TAG PROCESSING: Parse tag names using colon separator for multiple tags, resolve tag handles, validate existence. Support scalar and vector tag data.
4. DATA RECEPTION: Use ParCommGraph to receive tag data via receive_tag_values, unpack received tag values and associate with local entities, execute non-blocking point-to-point receives.

Data Received:

• Tag values: Scalar or vector data from sender component
• Entity mappings: Which entities the tag values belong to
• Tag metadata: Tag type, size, and component information
• Global IDs: For entity identification and matching

Parameters

[in]	pid	Application ID for receiver component
[in]	tag_storage_name	Tag name(s) to receive (colon-separated for multiple tags)
[in]	joint_communicator	MPI communicator spanning sender and receiver groups
[in]	context_id	Communication graph context ID

Note

Use cases: Receive solution data, get material properties/BCs, accept computed quantities

Warning

Communication graph must exist before calling (establish via mesh transfer or compute graph)

Returns

moab::MB_SUCCESS on success, moab::MB_FAILURE if graph missing or tags not found

Definition at line 3611 of file iMOAB.cpp.

{
    // Get application data and look up the communication graph for this context
    appData& data                               = context.appDatas[*pid];
    std::map< int, ParCommGraph* >::iterator mt = data.pgraph.find( *context_id );
    if( mt == data.pgraph.end() )
    {
        // Communication graph not found - this means mesh transfer hasn't been performed yet
        std::cout << " no par com graph for context_id:" << *context_id << " available contexts:";
        for( auto mit = data.pgraph.begin(); mit != data.pgraph.end(); mit++ )
            std::cout << "  " << mit->first;
        std::cout << "\n";
        return moab::MB_FAILURE;
    }
 
    // Extract communication graph and parallel communicator
    ParCommGraph* cgraph = mt->second;
    ParallelComm* pco    = context.appDatas[*pid].pcomm;
 
    // Handle Fortran-to-C MPI handle conversion if needed
    MPI_Comm global = ( data.is_fortran ? MPI_Comm_f2c( *reinterpret_cast< MPI_Fint* >( joint_communicator ) )
                                        : *joint_communicator );
 
    // Determine target entities: vertices for point clouds, elements for regular meshes
    Range owned = ( data.point_cloud ? data.local_verts : data.owned_elems );
 
    // Handle coverage set entities for intersection applications
    // This covers cases where elements have been instantiated in coverage sets during intersection
    EntityHandle cover_set = cgraph->get_cover_set();
    if( 0 != cover_set ) MB_CHK_ERR( context.MBI->get_entities_by_dimension( cover_set, 2, owned ) );
 
#ifdef MOAB_HAVE_TEMPESTREMAP
    // Handle TempestRemap coverage mesh entities for intersection applications
    if( data.tempestData.remapper != nullptr )  // This is the case for intersection operations
    {
        cover_set = data.tempestData.remapper->GetMeshSet( Remapper::CoveringMesh );
        MB_CHK_ERR( context.MBI->get_entities_by_dimension( cover_set, 2, owned ) );
    }
#endif
 
    // Parse tag names from the input string (multiple tags separated by colons)
    std::string tag_name( tag_storage_name );
 
    // Parse multiple tag names separated by colons
    std::vector< std::string > tagNames;
    std::vector< Tag > tagHandles;
    std::string separator( ":" );
    split_tag_names( tag_name, separator, tagNames );
 
    // Resolve tag handles for each specified tag name
    for( size_t i = 0; i < tagNames.size(); i++ )
    {
        Tag tagHandle;
        ErrorCode rval = context.MBI->tag_get_handle( tagNames[i].c_str(), tagHandle );
        if( MB_SUCCESS != rval || nullptr == tagHandle )
        {
            MB_CHK_SET_ERR( moab::MB_FAILURE,
                            " can't get tag handle for tag named:" << tagNames[i].c_str() << " at index " << i );
        }
        tagHandles.push_back( tagHandle );
    }
 
#ifdef VERBOSE
    std::cout << pco->rank() << ". Looking to receive data for tags: " << tag_name
              << " and file set = " << ( data.file_set ) << "\n";
#endif
    // Execute the tag data reception using the communication graph
    // This receives tag values from sender processes and applies them to local entities
    // pco is needed for MOAB operations (unpacking), not for MPI communication
    MB_CHK_SET_ERR( cgraph->receive_tag_values( global, pco, owned, tagHandles ), "failed to receive tag values" );
 
#ifdef VERBOSE
    std::cout << pco->rank() << ". Looking to receive data for tags: " << tag_name << "\n";
#endif
 
    return moab::MB_SUCCESS;
}

References GlobalContext::appDatas, context, moab::Remapper::CoveringMesh, ErrorCode, appData::file_set, moab::ParCommGraph::get_cover_set(), moab::Interface::get_entities_by_dimension(), appData::is_fortran, appData::local_verts, MB_CHK_ERR, MB_CHK_SET_ERR, MB_SUCCESS, GlobalContext::MBI, appData::owned_elems, appData::pgraph, appData::point_cloud, moab::ParallelComm::rank(), moab::ParCommGraph::receive_tag_values(), split_tag_names(), and moab::Interface::tag_get_handle().

iMOAB_ReceiveMesh()

ErrCode iMOAB_ReceiveMesh	(	iMOAB_AppID	pid,
		MPI_Comm *	joint_communicator,
		MPI_Group *	sendingGroup,
		int *	scompid
	)

Receive mesh data on target component from sending component.

Implements receiver side of distributed mesh transfer protocol. Receives mesh entities and data from multiple sender processes, consolidates them into local mesh, and handles entity deduplication based on global IDs.

Algorithm Workflow:

1. COMMUNICATOR SETUP: Resolve MPI communicator and group handles (Fortran F2C conversions). Extract receiver group from MOAB ParallelComm. Create ParCommGraph with reverse mapping.
2. COMMUNICATION GRAPH RECEPTION: Receive communication graph via receive_comm_graph. Parse packed graph array to extract sender ranks for this receiver process.
3. SENDER IDENTIFICATION: Determine current receiver's rank index. Walk through packed graph array to find all sender ranks that contribute. Build senders_local list.
4. MESH RECEPTION: Invoke receive_mesh to pull mesh parts from identified sender processes. Deposit mesh entities into data.file_set (local mesh container).
5. POST-PROCESSING: Optional vertex merging for entities with identical GLOBAL_ID from different senders. Re-establish mesh information and update local mesh metadata.

Data Received:

• Mesh entities: elements and vertices with geometric data
• Connectivity: element-to-vertex relationships
• Tags: material properties, boundary conditions, custom data
• Global IDs: for entity identification and deduplication

Parameters

[in]	pid	Application ID for receiver component
[in]	joint_communicator	MPI communicator spanning both sender and receiver groups
[in]	sendingGroup	MPI group of sending processes
[in]	scompid	Sender component ID (key for communication graph storage)

Note

Communication: Collective (receive graph) and point-to-point (receive mesh data)

Warning

Vertex merging only performed if receiving from 2+ senders

Returns

moab::MB_SUCCESS on success, moab::MB_FAILURE on error

Definition at line 3287 of file iMOAB.cpp.

{
    // Validate input parameters to ensure they are not null pointers
    assert( joint_communicator != nullptr );
    assert( sendingGroup != nullptr );
    assert( scompid != nullptr );
 
    ErrorCode rval;
    // Get application data and parallel communicator for this component
    appData& data          = context.appDatas[*pid];
    ParallelComm* pco      = context.appDatas[*pid].pcomm;
    MPI_Comm receive       = pco->comm();
    EntityHandle local_set = data.file_set;
 
    // Handle Fortran-to-C MPI handle conversion if needed
    MPI_Comm global = ( data.is_fortran ? MPI_Comm_f2c( *reinterpret_cast< MPI_Fint* >( joint_communicator ) )
                                        : *joint_communicator );
    MPI_Group sendGroup =
        ( data.is_fortran ? MPI_Group_f2c( *reinterpret_cast< MPI_Fint* >( sendingGroup ) ) : *sendingGroup );
 
    // Extract receiver group from the receiver communicator to understand receiver process topology
    MPI_Group receiverGroup;
    int ierr = MPI_Comm_group( receive, &receiverGroup );CHK_MPI_ERR( ierr );
 
    // Create communication graph with reverse mapping (sender->receiver)
    // This graph will manage the reception of mesh data from sender processes
    ParCommGraph* cgraph =
        new ParCommGraph( global, sendGroup, receiverGroup, *scompid, context.appDatas[*pid].global_id );
 
    // Store the communication graph in the application's graph map for later use
    // The key is the sender component ID (*scompid)
    context.appDatas[*pid].pgraph[*scompid] = cgraph;
 
    // Get current receiver rank for process identification
    int receiver_rank = -1;
    MPI_Comm_rank( receive, &receiver_rank );
 
    // Receive the communication graph from sender processes
    // This graph tells this receiver which senders will send it data
    std::vector< int > pack_array;
    MB_CHK_ERR( cgraph->receive_comm_graph( global, pco, pack_array ) );
 
    // Determine this receiver's index within the receiver group
    int current_receiver = cgraph->receiver( receiver_rank );
 
    // Parse the packed communication graph to find which senders contribute to this receiver
    std::vector< int > senders_local;
    size_t n = 0;
 
    // Walk through the packed graph array to find sender ranks for this receiver
    while( n < pack_array.size() )
    {
        if( current_receiver == pack_array[n] )
        {
            // Found this receiver's entry - extract all contributing sender ranks
            for( int j = 0; j < pack_array[n + 1]; j++ )
            {
                senders_local.push_back( pack_array[n + 2 + j] );
            }
            break;
        }
        // Move to next receiver's entry in the packed array
        n = n + 2 + pack_array[n + 1];
    }
 
#ifdef VERBOSE
    std::cout << " receiver " << current_receiver << " at rank " << receiver_rank << " will receive from "
              << senders_local.size() << " tasks: ";
 
    for( int k = 0; k < (int)senders_local.size(); k++ )
    {
        std::cout << " " << senders_local[k];
    }
 
    std::cout << "\n";
#endif
 
    // Validate that we have senders contributing to this receiver
    if( senders_local.empty() )
    {
        std::cout << " we do not have any senders for receiver rank " << receiver_rank << "\n";
    }
 
    // Receive mesh data from all identified sender processes
    // This unpacks mesh entities and deposits them into the local mesh set
    MB_CHK_ERR( cgraph->receive_mesh( global, pco, local_set, senders_local ) );
 
    // Post-process received mesh data to handle potential vertex duplication
    // Vertices with identical GLOBAL_ID from different senders need to be merged
    Tag idtag;
    MB_CHK_ERR( context.MBI->tag_get_handle( "GLOBAL_ID", idtag ) );
 
    // Get all entities in the local mesh set
    Range local_ents;
    MB_CHK_ERR( context.MBI->get_entities_by_handle( local_set, local_ents ) );
 
    // Only perform vertex merging for regular meshes (not point clouds)
    if( !local_ents.all_of_type( MBVERTEX ) )
    {
        // Merge vertices only if we received data from multiple senders
        if( (int)senders_local.size() >= 2 )  // Need to remove duplicate vertices
        {
 
            // Separate vertices from elements for processing
            Range local_verts = local_ents.subset_by_type( MBVERTEX );
            Range local_elems = subtract( local_ents, local_verts );
 
            // Temporarily remove vertices from local set for merging
            MB_CHK_ERR( context.MBI->remove_entities( local_set, local_verts ) );
 
#ifdef VERBOSE
            std::cout << "current_receiver " << current_receiver << " local verts: " << local_verts.size() << "\n";
#endif
            // Merge vertices with identical GLOBAL_ID values
            MergeMesh mm( context.MBI );
            MB_CHK_ERR( mm.merge_using_integer_tag( local_verts, idtag ) );
 
            // Get the merged vertices back from element connectivity
            Range new_verts;
            MB_CHK_ERR( context.MBI->get_connectivity( local_elems, new_verts ) );
 
#ifdef VERBOSE
            std::cout << "after merging: new verts: " << new_verts.size() << "\n";
#endif
            // Add the merged vertices back to the local set
            MB_CHK_ERR( context.MBI->add_entities( local_set, new_verts ) );
        }
    }
    else
        // Mark this as a point cloud if we only have vertices
        data.point_cloud = true;
 
    if( !data.point_cloud )
    {
        // For regular meshes, resolve shared entities (vertices) across process boundaries
        MB_CHK_ERR( pco->resolve_shared_ents( local_set, -1, -1, &idtag ) );
    }
    else
    {
        // For point clouds, set partition tag to current rank for visualization
        Tag densePartTag;
        rval = context.MBI->tag_get_handle( "partition", densePartTag );
        if( nullptr != densePartTag && MB_SUCCESS == rval )
        {
            Range local_verts;
            MB_CHK_ERR( context.MBI->get_entities_by_dimension( local_set, 0, local_verts ) );
            std::vector< int > vals;
            int rank = pco->rank();
            vals.resize( local_verts.size(), rank );
            MB_CHK_ERR( context.MBI->tag_set_data( densePartTag, local_verts, &vals[0] ) );
        }
    }
    // Set the parallel partition tag to identify which process owns this mesh set
    Tag part_tag;
    int dum_id = -1;
    rval       = context.MBI->tag_get_handle( "PARALLEL_PARTITION", 1, MB_TYPE_INTEGER, part_tag,
                                              MB_TAG_CREAT | MB_TAG_SPARSE, &dum_id );
 
    if( part_tag == nullptr || ( ( rval != MB_SUCCESS ) && ( rval != MB_ALREADY_ALLOCATED ) ) )
    {
        std::cout << " can't get par part tag.\n";
        return moab::MB_FAILURE;
    }
 
    // Tag the local set with the current process rank
    int rank = pco->rank();
    MB_CHK_ERR( context.MBI->tag_set_data( part_tag, &local_set, 1, &rank ) );
 
    // Ensure that the GLOBAL_ID tag is properly defined for entity identification
    int tagtype = 0;  // dense, integer
    int numco   = 1;  // size
    int tagindex;     // not used
    MB_CHK_ERR( iMOAB_DefineTagStorage( pid, "GLOBAL_ID", &tagtype, &numco, &tagindex ) );
 
    // Update mesh information with current data statistics
    MB_CHK_ERR( iMOAB_UpdateMeshInfo( pid ) );
 
    // Clean up temporary MPI group to prevent memory leaks
    MPI_Group_free( &receiverGroup );
 
    return moab::MB_SUCCESS;
}

References moab::Interface::add_entities(), moab::Range::all_of_type(), GlobalContext::appDatas, CHK_MPI_ERR, moab::ParallelComm::comm(), context, ErrorCode, appData::file_set, moab::Interface::get_connectivity(), moab::Interface::get_entities_by_dimension(), moab::Interface::get_entities_by_handle(), iMOAB_DefineTagStorage(), iMOAB_UpdateMeshInfo(), appData::is_fortran, MB_ALREADY_ALLOCATED, MB_CHK_ERR, MB_SUCCESS, MB_TAG_CREAT, MB_TAG_SPARSE, MB_TYPE_INTEGER, GlobalContext::MBI, MBVERTEX, moab::MergeMesh::merge_using_integer_tag(), appData::point_cloud, moab::ParallelComm::rank(), moab::ParCommGraph::receive_comm_graph(), moab::ParCommGraph::receive_mesh(), moab::ParCommGraph::receiver(), moab::Interface::remove_entities(), moab::ParallelComm::resolve_shared_ents(), moab::Range::size(), moab::Range::subset_by_type(), moab::subtract(), moab::Interface::tag_get_handle(), and moab::Interface::tag_set_data().

iMOAB_SendElementTag()

ErrCode iMOAB_SendElementTag	(	iMOAB_AppID	pid,
		const iMOAB_String	tag_storage_name,
		MPI_Comm *	joint_communicator,
		int *	context_id
	)

Send element tag data from calling component to another component.

Transfers tag data (scalar or vector values) associated with mesh elements from one component to another using pre-established communication graph. Transfer based on communication patterns determined by previous mesh transfer or communication graph computation operations.

Algorithm Workflow:

1. COMMUNICATION GRAPH VALIDATION: Look up ParCommGraph for specified context_id, validate existence and initialization, extract graph and parallel communicator.
2. ENTITY SELECTION: Determine target entities based on component type (point cloud: local_verts, regular mesh: owned_elems). Handle special cases (TempestRemap coverage mesh, intersection coverage sets).
3. TAG PROCESSING: Parse tag names using colon separator for multiple tags, resolve tag handles, validate existence. Support scalar and vector tag data.
4. DATA TRANSFER: Use ParCommGraph to determine routing, pack tag values associated with selected entities, execute non-blocking point-to-point transfers via send_tag_values.

Data Transferred:

• Tag values: Scalar or vector data associated with mesh entities
• Entity mappings: Which entities the tag values belong to
• Tag metadata: Tag type, size, and component information
• Global IDs: For entity identification across processes

Parameters

[in]	pid	Application ID for sender component
[in]	tag_storage_name	Tag name(s) to send (colon-separated for multiple tags)
[in]	joint_communicator	MPI communicator spanning sender and receiver groups
[in]	context_id	Communication graph context ID

Note

Use cases: Transfer solution data, send material properties/BCs, exchange computed quantities

Warning

Communication graph must exist before calling (establish via mesh transfer or compute graph)

Returns

moab::MB_SUCCESS on success, moab::MB_FAILURE if graph missing or tags not found

Definition at line 3504 of file iMOAB.cpp.

{
    // Get application data and look up the communication graph for this context
    appData& data                               = context.appDatas[*pid];
    std::map< int, ParCommGraph* >::iterator mt = data.pgraph.find( *context_id );
    if( mt == data.pgraph.end() )
    {
        // Communication graph not found - this means mesh transfer hasn't been performed yet
        std::cout << " no par com graph for context_id:" << *context_id << " available contexts:";
        for( auto mit = data.pgraph.begin(); mit != data.pgraph.end(); mit++ )
            std::cout << "  " << mit->first;
        std::cout << "\n";
        return moab::MB_FAILURE;
    }
 
    // Extract communication graph and parallel communicator
    ParCommGraph* cgraph = mt->second;
    ParallelComm* pco    = context.appDatas[*pid].pcomm;
 
    // Handle Fortran-to-C MPI handle conversion if needed
    MPI_Comm global = ( data.is_fortran ? MPI_Comm_f2c( *reinterpret_cast< MPI_Fint* >( joint_communicator ) )
                                        : *joint_communicator );
 
    // Determine target entities: vertices for point clouds, elements for regular meshes
    Range owned = ( data.point_cloud ? data.local_verts : data.owned_elems );
 
#ifdef MOAB_HAVE_TEMPESTREMAP
    // Handle TempestRemap coverage mesh entities for intersection applications
    if( data.tempestData.remapper != nullptr )  // This is the case for intersection operations
    {
        EntityHandle cover_set = data.tempestData.remapper->GetMeshSet( Remapper::CoveringMesh );
        MB_CHK_ERR( context.MBI->get_entities_by_dimension( cover_set, 2, owned ) );
    }
#else
    // Handle coverage set entities for intersection applications (non-TempestRemap case)
    // This covers cases where elements have been instantiated in coverage sets during intersection
    EntityHandle cover_set = cgraph->get_cover_set();  // Non-null only for intersection applications
    if( 0 != cover_set ) MB_CHK_ERR( context.MBI->get_entities_by_dimension( cover_set, 2, owned ) );
#endif
 
    // Parse tag names from the input string (multiple tags separated by colons)
    std::string tag_name( tag_storage_name );
 
    // Parse multiple tag names separated by colons
    std::vector< std::string > tagNames;
    std::vector< Tag > tagHandles;
    std::string separator( ":" );
    split_tag_names( tag_name, separator, tagNames );
 
    // Resolve tag handles for each specified tag name
    for( size_t i = 0; i < tagNames.size(); i++ )
    {
        Tag tagHandle;
        ErrorCode rval = context.MBI->tag_get_handle( tagNames[i].c_str(), tagHandle );
        if( MB_SUCCESS != rval || nullptr == tagHandle )
        {
            MB_CHK_SET_ERR( moab::MB_FAILURE,
                            "can't get tag handle with name: " << tagNames[i].c_str() << " at index " << i );
        }
        tagHandles.push_back( tagHandle );
    }
 
    // Execute the tag data transfer using the communication graph
    // This packs tag values and sends them to appropriate receiver processes
    // pco is needed for MOAB operations (packing), not for MPI communication
    MB_CHK_ERR( cgraph->send_tag_values( global, pco, owned, tagHandles ) );
 
    return moab::MB_SUCCESS;
}

References GlobalContext::appDatas, context, moab::Remapper::CoveringMesh, ErrorCode, moab::ParCommGraph::get_cover_set(), moab::Interface::get_entities_by_dimension(), appData::is_fortran, appData::local_verts, MB_CHK_ERR, MB_CHK_SET_ERR, MB_SUCCESS, GlobalContext::MBI, appData::owned_elems, appData::pgraph, appData::point_cloud, moab::ParCommGraph::send_tag_values(), split_tag_names(), and moab::Interface::tag_get_handle().

iMOAB_SendMesh()

ErrCode iMOAB_SendMesh	(	iMOAB_AppID	pid,
		MPI_Comm *	joint_communicator,
		MPI_Group *	receivingGroup,
		int *	rcompid,
		int *	method
	)

Transfer mesh data from sender component to receiver component.

Implements distributed mesh transfer protocol that moves mesh entities (elements, vertices, and associated data) from one set of MPI processes to another. Transfer is orchestrated through a communication graph that maps sender processes to receiver processes based on partitioning strategy.

Algorithm Workflow:

1. COMMUNICATOR SETUP: Resolve MPI communicator and group handles (handles Fortran F2C conversions). Extract sender group from MOAB ParallelComm. Create ParCommGraph for sender→receiver mapping.
2. ENTITY SELECTION: Determine owned entities to transfer: owned_elems or local_verts (point cloud mode). Store ParCommGraph in context.appDatas[pid].pgraph[rcompid].
3. PARTITIONING STRATEGY:
   • method == 0 (TRIVIAL): Gather global entity counts, compute block distribution, broadcast via send_graph
   • method != 0 (COMPUTED): Use graph/geometric partitioning considering connectivity and spatial distribution
4. MESH TRANSFER: Pack mesh entities and data, use ParCommGraph to route to receivers via send_mesh_parts

Data Transferred:

• Mesh entities: elements (triangles, quads, etc.) and vertices
• Geometric data: vertex coordinates, element connectivity
• Metadata: material sets, boundary conditions, tags
• Global IDs: for entity identification across processes

Parameters

[in]	pid	Application ID for sender component
[in]	joint_communicator	MPI communicator spanning both sender and receiver groups
[in]	receivingGroup	MPI group of receiving processes
[in]	rcompid	Receiver component ID (key for communication graph storage)
[in]	method	Partitioning method: 0=trivial, !=0=computed

Note

Communication: Collective (MPI_Allgather) and point-to-point (non-blocking sends)

Warning

Cleans up MPI groups on early return to avoid resource leaks

Returns

moab::MB_SUCCESS on success, moab::MB_FAILURE on error

Definition at line 3140 of file iMOAB.cpp.

{
    // Validate input parameters to ensure they are not null pointers
    assert( joint_communicator != nullptr );
    assert( receivingGroup != nullptr );
    assert( rcompid != nullptr );
 
    int ierr;
    // Get application data and parallel communicator for this component
    appData& data     = context.appDatas[*pid];
    ParallelComm* pco = context.appDatas[*pid].pcomm;
 
    // Handle Fortran-to-C MPI handle conversion if needed
    // Fortran passes MPI handles as integers, C++ expects MPI_Comm/MPI_Group objects
    MPI_Comm global = ( data.is_fortran ? MPI_Comm_f2c( *reinterpret_cast< MPI_Fint* >( joint_communicator ) )
                                        : *joint_communicator );
    MPI_Group recvGroup =
        ( data.is_fortran ? MPI_Group_f2c( *reinterpret_cast< MPI_Fint* >( receivingGroup ) ) : *receivingGroup );
 
    // Extract sender communicator from MOAB's parallel communicator
    // This represents the group of processes that will send mesh data
    MPI_Comm sender = pco->comm();
    // Extract sender group from the sender communicator to understand sender process topology
    MPI_Group senderGroup;
    ierr = MPI_Comm_group( sender, &senderGroup );
    if( ierr != 0 ) return moab::MB_FAILURE;
 
    // Create communication graph to manage sender->receiver process mapping
    // This graph will determine which entities go to which receiver processes
    ParCommGraph* cgraph =
        new ParCommGraph( global, senderGroup, recvGroup, context.appDatas[*pid].global_id, *rcompid );
 
    // Store the communication graph in the application's graph map for later use
    // The key is the receiver component ID (*rcompid)
    context.appDatas[*pid].pgraph[*rcompid] = cgraph;
 
    // Get current sender rank for potential debugging/logging
    int sender_rank = -1;
    MPI_Comm_rank( sender, &sender_rank );
 
    // Determine which entities to send: primary elements or vertices (for point clouds)
    std::vector< int > number_elems_per_part;
    Range owned = context.appDatas[*pid].owned_elems;
    if( owned.size() == 0 )
    {
        // If no elements, this must be a point cloud - send vertices instead
        owned = context.appDatas[*pid].local_verts;
    }
 
    // Choose partitioning strategy based on method parameter
    if( *method == 0 )  // Trivial partitioning: simple block distribution
    {
        // Count local entities on this sender process
        int local_owned_elem = (int)owned.size();
        int size             = pco->size();
        int rank             = pco->rank();
 
        // Prepare array to hold entity counts from all sender processes
        number_elems_per_part.resize( size );
        number_elems_per_part[rank] = local_owned_elem;
        // Gather entity counts from all sender processes using MPI_Allgather
        // This allows each sender to know the total entity count for load balancing
#if ( MPI_VERSION >= 2 )
        // Use "in place" option for efficiency (MPI 2.0+)
        ierr = MPI_Allgather( MPI_IN_PLACE, 0, MPI_DATATYPE_NULL, &number_elems_per_part[0], 1, MPI_INT, sender );
#else
        // Fallback for older MPI versions
        {
            std::vector< int > all_tmp( size );
            ierr = MPI_Allgather( &number_elems_per_part[rank], 1, MPI_INT, &all_tmp[0], 1, MPI_INT, sender );
            number_elems_per_part = all_tmp;
        }
#endif
 
        if( ierr != 0 )
        {
            return moab::MB_FAILURE;
        }
 
        // Compute trivial partition: each receiver gets roughly equal share of entities
        // This creates a simple block distribution based on entity counts
        MB_CHK_ERR( cgraph->compute_trivial_partition( number_elems_per_part ) );
 
        // Send the partition mapping to all receiver processes
        // This tells receivers which senders will send them data
        MB_CHK_ERR( cgraph->send_graph( global ) );
    }
    else  // Advanced partitioning: graph-based or geometric partitioning
    {
        // Use sophisticated partitioning that considers entity connectivity or spatial distribution
        // This can lead to better load balancing and reduced communication
        MB_CHK_ERR( cgraph->compute_partition( pco, owned, *method ) );
 
        // Send the computed partition mapping to receiver processes
        // This includes more complex sender->receiver mappings than trivial partition
        MB_CHK_ERR( cgraph->send_graph_partition( pco, global ) );
    }
    // Execute the actual mesh transfer based on the communication graph
    // This packs mesh entities and sends them to appropriate receiver processes
    // pco is needed for MOAB operations (packing), not for MPI communication
    MB_CHK_ERR( cgraph->send_mesh_parts( global, pco, owned ) );
 
    // Clean up temporary MPI group to prevent memory leaks
    MPI_Group_free( &senderGroup );
    return moab::MB_SUCCESS;
}

References GlobalContext::appDatas, moab::ParallelComm::comm(), moab::ParCommGraph::compute_partition(), moab::ParCommGraph::compute_trivial_partition(), context, appData::is_fortran, MB_CHK_ERR, MB_SUCCESS, moab::ParallelComm::rank(), moab::ParCommGraph::send_graph(), moab::ParCommGraph::send_graph_partition(), moab::ParCommGraph::send_mesh_parts(), moab::Range::size(), and moab::ParallelComm::size().