Mesh Oriented datABase  (version 5.6.0)
An array-based unstructured mesh library
Classes | Public Member Functions | Static Public Member Functions | Static Public Attributes | Private Member Functions | Private Attributes | Friends | List of all members
moab::ParallelComm Class Reference

Parallel communications in MOAB. More...

#include <ParallelComm.hpp>

+ Collaboration diagram for moab::ParallelComm:

Classes

class  Buffer
 
struct  SharedEntityData
 

Public Member Functions

 ParallelComm (Interface *impl, MPI_Comm comm, int *pcomm_id_out=0)
 constructor More...
 
 ParallelComm (Interface *impl, std::vector< unsigned char > &tmp_buff, MPI_Comm comm, int *pcomm_id_out=0)
 constructor taking packed buffer, for testing More...
 
int get_id () const
 Get ID used to reference this PCOMM instance. More...
 
 ~ParallelComm ()
 destructor More...
 
ErrorCode assign_global_ids (EntityHandle this_set, const int dimension, const int start_id=1, const bool largest_dim_only=true, const bool parallel=true, const bool owned_only=false)
 assign a global id space, for largest-dimension or all entities (and in either case for vertices too) More...
 
ErrorCode assign_global_ids (Range entities[], const int dimension, const int start_id, const bool parallel, const bool owned_only)
 assign a global id space, for largest-dimension or all entities (and in either case for vertices too) More...
 
ErrorCode check_global_ids (EntityHandle this_set, const int dimension, const int start_id=1, const bool largest_dim_only=true, const bool parallel=true, const bool owned_only=false)
 check for global ids; based only on tag handle being there or not; if it's not there, create them for the specified dimensions More...
 
ErrorCode send_entities (const int to_proc, Range &orig_ents, const bool adjs, const bool tags, const bool store_remote_handles, const bool is_iface, Range &final_ents, int &incoming1, int &incoming2, TupleList &entprocs, std::vector< MPI_Request > &recv_remoteh_reqs, bool wait_all=true)
 send entities to another processor, optionally waiting until it's done More...
 
ErrorCode send_entities (std::vector< unsigned int > &send_procs, std::vector< Range * > &send_ents, int &incoming1, int &incoming2, const bool store_remote_handles)
 
ErrorCode recv_entities (const int from_proc, const bool store_remote_handles, const bool is_iface, Range &final_ents, int &incomming1, int &incoming2, std::vector< std::vector< EntityHandle > > &L1hloc, std::vector< std::vector< EntityHandle > > &L1hrem, std::vector< std::vector< int > > &L1p, std::vector< EntityHandle > &L2hloc, std::vector< EntityHandle > &L2hrem, std::vector< unsigned int > &L2p, std::vector< MPI_Request > &recv_remoteh_reqs, bool wait_all=true)
 Receive entities from another processor, optionally waiting until it's done. More...
 
ErrorCode recv_entities (std::set< unsigned int > &recv_procs, int incoming1, int incoming2, const bool store_remote_handles, const bool migrate=false)
 
ErrorCode recv_messages (const int from_proc, const bool store_remote_handles, const bool is_iface, Range &final_ents, int &incoming1, int &incoming2, std::vector< std::vector< EntityHandle > > &L1hloc, std::vector< std::vector< EntityHandle > > &L1hrem, std::vector< std::vector< int > > &L1p, std::vector< EntityHandle > &L2hloc, std::vector< EntityHandle > &L2hrem, std::vector< unsigned int > &L2p, std::vector< MPI_Request > &recv_remoteh_reqs)
 Receive messages from another processor in while loop. More...
 
ErrorCode recv_remote_handle_messages (const int from_proc, int &incoming2, std::vector< EntityHandle > &L2hloc, std::vector< EntityHandle > &L2hrem, std::vector< unsigned int > &L2p, std::vector< MPI_Request > &recv_remoteh_reqs)
 
ErrorCode exchange_ghost_cells (int ghost_dim, int bridge_dim, int num_layers, int addl_ents, bool store_remote_handles, bool wait_all=true, EntityHandle *file_set=NULL)
 Exchange ghost cells with neighboring procs Neighboring processors are those sharing an interface with this processor. All entities of dimension ghost_dim within num_layers of interface, measured going through bridge_dim, are exchanged. See MeshTopoUtil::get_bridge_adjacencies for description of bridge adjacencies. If wait_all is false and store_remote_handles is true, MPI_Request objects are available in the sendReqs[2*MAX_SHARING_PROCS] member array, with inactive requests marked as MPI_REQUEST_NULL. If store_remote_handles or wait_all is false, this function returns after all entities have been received and processed. More...
 
ErrorCode post_irecv (std::vector< unsigned int > &exchange_procs)
 Post "MPI_Irecv" before meshing. More...
 
ErrorCode post_irecv (std::vector< unsigned int > &shared_procs, std::set< unsigned int > &recv_procs)
 
ErrorCode exchange_owned_meshs (std::vector< unsigned int > &exchange_procs, std::vector< Range * > &exchange_ents, std::vector< MPI_Request > &recv_ent_reqs, std::vector< MPI_Request > &recv_remoteh_reqs, bool store_remote_handles, bool wait_all=true, bool migrate=false, int dim=0)
 Exchange owned mesh for input mesh entities and sets This function should be called collectively over the communicator for this ParallelComm. If this version is called, all shared exchanged entities should have a value for this tag (or the tag should have a default value). More...
 
ErrorCode exchange_owned_mesh (std::vector< unsigned int > &exchange_procs, std::vector< Range * > &exchange_ents, std::vector< MPI_Request > &recv_ent_reqs, std::vector< MPI_Request > &recv_remoteh_reqs, const bool recv_posted, bool store_remote_handles, bool wait_all, bool migrate=false)
 Exchange owned mesh for input mesh entities and sets This function is called twice by exchange_owned_meshs to exchange entities before sets. More...
 
ErrorCode exchange_tags (const std::vector< Tag > &src_tags, const std::vector< Tag > &dst_tags, const Range &entities)
 Exchange tags for all shared and ghosted entities This function should be called collectively over the communicator for this ParallelComm. If this version is called, all ghosted/shared entities should have a value for this tag (or the tag should have a default value). If the entities vector is empty, all shared entities participate in the exchange. If a proc has no owned entities this function must still be called since it is collective. More...
 
ErrorCode exchange_tags (const char *tag_name, const Range &entities)
 Exchange tags for all shared and ghosted entities This function should be called collectively over the communicator for this ParallelComm. If the entities vector is empty, all shared entities participate in the exchange. If a proc has no owned entities this function must still be called since it is collective. More...
 
ErrorCode exchange_tags (Tag tagh, const Range &entities)
 Exchange tags for all shared and ghosted entities This function should be called collectively over the communicator for this ParallelComm. If the entities vector is empty, all shared entities participate in the exchange. If a proc has no owned entities this function must still be called since it is collective. More...
 
ErrorCode reduce_tags (const std::vector< Tag > &src_tags, const std::vector< Tag > &dst_tags, const MPI_Op mpi_op, const Range &entities)
 Perform data reduction operation for all shared and ghosted entities This function should be called collectively over the communicator for this ParallelComm. If this version is called, all ghosted/shared entities should have a value for this tag (or the tag should have a default value). Operation is any MPI_Op, with result stored in destination tag. More...
 
ErrorCode reduce_tags (const char *tag_name, const MPI_Op mpi_op, const Range &entities)
 Perform data reduction operation for all shared and ghosted entities Same as std::vector variant except for one tag specified by name. More...
 
ErrorCode reduce_tags (Tag tag_handle, const MPI_Op mpi_op, const Range &entities)
 Perform data reduction operation for all shared and ghosted entities Same as std::vector variant except for one tag specified by handle. More...
 
ErrorCode broadcast_entities (const int from_proc, Range &entities, const bool adjacencies=false, const bool tags=true)
 Broadcast all entities resident on from_proc to other processors This function assumes remote handles are not being stored, since (usually) every processor will know about the whole mesh. More...
 
ErrorCode scatter_entities (const int from_proc, std::vector< Range > &entities, const bool adjacencies=false, const bool tags=true)
 Scatter entities on from_proc to other processors This function assumes remote handles are not being stored, since (usually) every processor will know about the whole mesh. More...
 
ErrorCode send_recv_entities (std::vector< int > &send_procs, std::vector< std::vector< int > > &msgsizes, std::vector< std::vector< EntityHandle > > &senddata, std::vector< std::vector< EntityHandle > > &recvdata)
 Send and receives data from a set of processors. More...
 
ErrorCode update_remote_data (EntityHandle entity, std::vector< int > &procs, std::vector< EntityHandle > &handles)
 
ErrorCode get_remote_handles (EntityHandle *local_vec, EntityHandle *rem_vec, int num_ents, int to_proc)
 
ErrorCode resolve_shared_ents (EntityHandle this_set, Range &proc_ents, int resolve_dim=-1, int shared_dim=-1, Range *skin_ents=NULL, const Tag *id_tag=0)
 Resolve shared entities between processors. More...
 
ErrorCode resolve_shared_ents (EntityHandle this_set, int resolve_dim=3, int shared_dim=-1, const Tag *id_tag=0)
 Resolve shared entities between processors. More...
 
ErrorCode resolve_shared_sets (EntityHandle this_set, const Tag *id_tag=0)
 
ErrorCode resolve_shared_sets (Range &candidate_sets, Tag id_tag)
 
ErrorCode augment_default_sets_with_ghosts (EntityHandle file_set)
 
ErrorCode get_pstatus (EntityHandle entity, unsigned char &pstatus_val)
 Get parallel status of an entity Returns the parallel status of an entity. More...
 
ErrorCode get_pstatus_entities (int dim, unsigned char pstatus_val, Range &pstatus_ents)
 Get entities with the given pstatus bit(s) set Returns any entities whose pstatus tag value v satisfies (v & pstatus_val) More...
 
ErrorCode get_owner (EntityHandle entity, int &owner)
 Return the rank of the entity owner. More...
 
ErrorCode get_owner_handle (EntityHandle entity, int &owner, EntityHandle &handle)
 Return the owner processor and handle of a given entity. More...
 
ErrorCode get_sharing_data (const EntityHandle entity, int *ps, EntityHandle *hs, unsigned char &pstat, unsigned int &num_ps)
 Get the shared processors/handles for an entity Get the shared processors/handles for an entity. Arrays must be large enough to receive data for all sharing procs. Does not include this proc if only shared with one other proc. More...
 
ErrorCode get_sharing_data (const EntityHandle entity, int *ps, EntityHandle *hs, unsigned char &pstat, int &num_ps)
 Get the shared processors/handles for an entity Same as other version but with int num_ps. More...
 
ErrorCode get_sharing_data (const EntityHandle *entities, int num_entities, std::set< int > &procs, int op=Interface::INTERSECT)
 Get the intersection or union of all sharing processors Get the intersection or union of all sharing processors. Processor set is cleared as part of this function. More...
 
ErrorCode get_sharing_data (const Range &entities, std::set< int > &procs, int op=Interface::INTERSECT)
 Get the intersection or union of all sharing processors Same as previous variant but with range as input. More...
 
ErrorCode get_shared_entities (int other_proc, Range &shared_ents, int dim=-1, const bool iface=false, const bool owned_filter=false)
 Get shared entities of specified dimension If other_proc is -1, any shared entities are returned. If dim is -1, entities of all dimensions on interface are returned. More...
 
ErrorCode get_interface_procs (std::set< unsigned int > &iface_procs, const bool get_buffs=false)
 get processors with which this processor shares an interface More...
 
ErrorCode get_comm_procs (std::set< unsigned int > &procs)
 get processors with which this processor communicates More...
 
ErrorCode get_entityset_procs (EntityHandle entity_set, std::vector< unsigned > &ranks) const
 Get array of process IDs sharing a set. Returns zero and passes back NULL if set is not shared. More...
 
ErrorCode get_entityset_owner (EntityHandle entity_set, unsigned &owner_rank, EntityHandle *remote_handle=0) const
 Get rank of the owner of a shared set. Returns this proc if set is not shared. Optionally returns handle on owning process for shared set. More...
 
ErrorCode get_entityset_local_handle (unsigned owning_rank, EntityHandle remote_handle, EntityHandle &local_handle) const
 Given set owner and handle on owner, find local set handle. More...
 
ErrorCode get_shared_sets (Range &result) const
 Get all shared sets. More...
 
ErrorCode get_entityset_owners (std::vector< unsigned > &ranks) const
 Get ranks of all processes that own at least one set that is shared with this process. Will include the rank of this process if this process owns any shared set. More...
 
ErrorCode get_owned_sets (unsigned owning_rank, Range &sets_out) const
 Get shared sets owned by process with specified rank. More...
 
const ProcConfig & proc_config () const
 Get proc config for this communication object. More...
 
ProcConfig & proc_config ()
 Get proc config for this communication object. More...
 
unsigned rank () const
 
unsigned size () const
 
MPI_Comm comm () const
 
ErrorCode get_shared_proc_tags (Tag &sharedp_tag, Tag &sharedps_tag, Tag &sharedh_tag, Tag &sharedhs_tag, Tag &pstatus_tag)
 return the tags used to indicate shared procs and handles More...
 
Range & partition_sets ()
 return partition, interface set ranges More...
 
const Range & partition_sets () const
 
Range & interface_sets ()
 
const Range & interface_sets () const
 
Tag sharedp_tag ()
 return sharedp tag More...
 
Tag sharedps_tag ()
 return sharedps tag More...
 
Tag sharedh_tag ()
 return sharedh tag More...
 
Tag sharedhs_tag ()
 return sharedhs tag More...
 
Tag pstatus_tag ()
 return pstatus tag More...
 
Tag partition_tag ()
 return partitions set tag More...
 
Tag part_tag ()
 
void print_pstatus (unsigned char pstat, std::string &ostr)
 print contents of pstatus value in human-readable form More...
 
void print_pstatus (unsigned char pstat)
 print contents of pstatus value in human-readable form to std::cut More...
 
ErrorCode get_part_entities (Range &ents, int dim=-1)
 return all the entities in parts owned locally More...
 
EntityHandle get_partitioning () const
 
ErrorCode set_partitioning (EntityHandle h)
 
ErrorCode get_global_part_count (int &count_out) const
 
ErrorCode get_part_owner (int part_id, int &owner_out) const
 
ErrorCode get_part_id (EntityHandle part, int &id_out) const
 
ErrorCode get_part_handle (int id, EntityHandle &handle_out) const
 
ErrorCode create_part (EntityHandle &part_out)
 
ErrorCode destroy_part (EntityHandle part)
 
ErrorCode collective_sync_partition ()
 
ErrorCode get_part_neighbor_ids (EntityHandle part, int neighbors_out[MAX_SHARING_PROCS], int &num_neighbors_out)
 
ErrorCode get_interface_sets (EntityHandle part, Range &iface_sets_out, int *adj_part_id=0)
 
ErrorCode get_owning_part (EntityHandle entity, int &owning_part_id_out, EntityHandle *owning_handle=0)
 
ErrorCode get_sharing_parts (EntityHandle entity, int part_ids_out[MAX_SHARING_PROCS], int &num_part_ids_out, EntityHandle remote_handles[MAX_SHARING_PROCS]=0)
 
ErrorCode filter_pstatus (Range &ents, const unsigned char pstatus_val, const unsigned char op, int to_proc=-1, Range *returned_ents=NULL)
 
ErrorCode get_iface_entities (int other_proc, int dim, Range &iface_ents)
 Get entities on interfaces shared with another proc. More...
 
Interface * get_moab () const
 
ErrorCode clean_shared_tags (std::vector< Range * > &exchange_ents)
 
ErrorCode pack_buffer (Range &orig_ents, const bool adjacencies, const bool tags, const bool store_remote_handles, const int to_proc, Buffer *buff, TupleList *entprocs=NULL, Range *allsent=NULL)
 public 'cuz we want to unit test these externally More...
 
ErrorCode unpack_buffer (unsigned char *buff_ptr, const bool store_remote_handles, const int from_proc, const int ind, std::vector< std::vector< EntityHandle > > &L1hloc, std::vector< std::vector< EntityHandle > > &L1hrem, std::vector< std::vector< int > > &L1p, std::vector< EntityHandle > &L2hloc, std::vector< EntityHandle > &L2hrem, std::vector< unsigned int > &L2p, std::vector< EntityHandle > &new_ents, const bool created_iface=false)
 
ErrorCode pack_entities (Range &entities, Buffer *buff, const bool store_remote_handles, const int to_proc, const bool is_iface, TupleList *entprocs=NULL, Range *allsent=NULL)
 
ErrorCode unpack_entities (unsigned char *&buff_ptr, const bool store_remote_handles, const int from_ind, const bool is_iface, std::vector< std::vector< EntityHandle > > &L1hloc, std::vector< std::vector< EntityHandle > > &L1hrem, std::vector< std::vector< int > > &L1p, std::vector< EntityHandle > &L2hloc, std::vector< EntityHandle > &L2hrem, std::vector< unsigned int > &L2p, std::vector< EntityHandle > &new_ents, const bool created_iface=false)
 unpack entities in buff_ptr More...
 
ErrorCode check_all_shared_handles (bool print_em=false)
 Call exchange_all_shared_handles, then compare the results with tag data on local shared entities. More...
 
ErrorCode pack_shared_handles (std::vector< std::vector< SharedEntityData > > &send_data)
 
ErrorCode check_local_shared ()
 
ErrorCode check_my_shared_handles (std::vector< std::vector< SharedEntityData > > &shents, const char *prefix=NULL)
 
void set_rank (unsigned int r)
 set rank for this pcomm; USED FOR TESTING ONLY! More...
 
void set_size (unsigned int r)
 set rank for this pcomm; USED FOR TESTING ONLY! More...
 
int get_buffers (int to_proc, bool *is_new=NULL)
 get (and possibly allocate) buffers for messages to/from to_proc; returns index of to_proc in buffProcs vector; if is_new is non-NULL, sets to whether new buffer was allocated PUBLIC ONLY FOR TESTING! More...
 
const std::vector< unsigned int > & buff_procs () const
 get buff processor vector More...
 
ErrorCode unpack_remote_handles (unsigned int from_proc, unsigned char *&buff_ptr, std::vector< EntityHandle > &L2hloc, std::vector< EntityHandle > &L2hrem, std::vector< unsigned int > &L2p)
 
ErrorCode pack_remote_handles (std::vector< EntityHandle > &L1hloc, std::vector< EntityHandle > &L1hrem, std::vector< int > &procs, unsigned int to_proc, Buffer *buff)
 
ErrorCode create_interface_sets (std::map< std::vector< int >, std::vector< EntityHandle > > &proc_nvecs)
 
ErrorCode create_interface_sets (EntityHandle this_set, int resolve_dim, int shared_dim)
 
ErrorCode tag_shared_verts (TupleList &shared_ents, std::map< std::vector< int >, std::vector< EntityHandle > > &proc_nvecs, Range &proc_verts, unsigned int i_extra=1)
 
ErrorCode list_entities (const EntityHandle *ents, int num_ents)
 
ErrorCode list_entities (const Range &ents)
 
void set_send_request (int n_request)
 
void set_recv_request (int n_request)
 
void reset_all_buffers ()
 reset message buffers to their initial state More...
 
void set_debug_verbosity (int verb)
 set the verbosity level of output from this pcomm More...
 
int get_debug_verbosity ()
 get the verbosity level of output from this pcomm More...
 
ErrorCode gather_data (Range &gather_ents, Tag &tag_handle, Tag id_tag=0, EntityHandle gather_set=0, int root_proc_rank=0)
 
ErrorCode settle_intersection_points (Range &edges, Range &shared_edges_owned, std::vector< std::vector< EntityHandle > * > &extraNodesVec, double tolerance)
 
ErrorCode delete_entities (Range &to_delete)
 
ErrorCode correct_thin_ghost_layers ()
 

Static Public Member Functions

static ParallelComm * get_pcomm (Interface *impl, const int index)
 get the indexed pcomm object from the interface More...
 
static ParallelComm * get_pcomm (Interface *impl, EntityHandle partitioning, const MPI_Comm *comm=0)
 Get ParallelComm instance associated with partition handle Will create ParallelComm instance if a) one does not already exist and b) a valid value for MPI_Comm is passed. More...
 
static ErrorCode get_all_pcomm (Interface *impl, std::vector< ParallelComm * > &list)
 
static ErrorCode exchange_ghost_cells (ParallelComm **pc, unsigned int num_procs, int ghost_dim, int bridge_dim, int num_layers, int addl_ents, bool store_remote_handles, EntityHandle *file_sets=NULL)
 Static version of exchange_ghost_cells, exchanging info through buffers rather than messages. More...
 
static ErrorCode resolve_shared_ents (ParallelComm **pc, const unsigned int np, EntityHandle this_set, const int to_dim)
 
static Tag pcomm_tag (Interface *impl, bool create_if_missing=true)
 return pcomm tag; static because might not have a pcomm before going to look for one on the interface More...
 
static ErrorCode check_all_shared_handles (ParallelComm **pcs, int num_pcs)
 

Static Public Attributes

static unsigned char PROC_SHARED
 
static unsigned char PROC_OWNER
 
static const unsigned int INITIAL_BUFF_SIZE = 1024
 

Private Member Functions

ErrorCode reduce_void (int tag_data_type, const MPI_Op mpi_op, int num_ents, void *old_vals, void *new_vals)
 
template<class T >
ErrorCode reduce (const MPI_Op mpi_op, int num_ents, void *old_vals, void *new_vals)
 
void print_debug_isend (int from, int to, unsigned char *buff, int tag, int size)
 
void print_debug_irecv (int to, int from, unsigned char *buff, int size, int tag, int incoming)
 
void print_debug_recd (MPI_Status status)
 
void print_debug_waitany (std::vector< MPI_Request > &reqs, int tag, int proc)
 
void initialize ()
 
ErrorCode set_sharing_data (EntityHandle ent, unsigned char pstatus, int old_nump, int new_nump, int *ps, EntityHandle *hs)
 
ErrorCode check_clean_iface (Range &allsent)
 
void define_mpe ()
 
ErrorCode get_sent_ents (const bool is_iface, const int bridge_dim, const int ghost_dim, const int num_layers, const int addl_ents, Range *sent_ents, Range &allsent, TupleList &entprocs)
 
ErrorCode set_pstatus_entities (Range &pstatus_ents, unsigned char pstatus_val, bool lower_dim_ents=false, bool verts_too=true, int operation=Interface::UNION)
 Set pstatus values on entities. More...
 
ErrorCode set_pstatus_entities (EntityHandle *pstatus_ents, int num_ents, unsigned char pstatus_val, bool lower_dim_ents=false, bool verts_too=true, int operation=Interface::UNION)
 Set pstatus values on entities (vector-based function) More...
 
int estimate_ents_buffer_size (Range &entities, const bool store_remote_handles)
 estimate size required to pack entities More...
 
int estimate_sets_buffer_size (Range &entities, const bool store_remote_handles)
 estimate size required to pack sets More...
 
ErrorCode send_buffer (const unsigned int to_proc, Buffer *send_buff, const int msg_tag, MPI_Request &send_req, MPI_Request &ack_recv_req, int *ack_buff, int &this_incoming, int next_mesg_tag=-1, Buffer *next_recv_buff=NULL, MPI_Request *next_recv_req=NULL, int *next_incoming=NULL)
 send the indicated buffer, possibly sending size first More...
 
ErrorCode recv_buffer (int mesg_tag_expected, const MPI_Status &mpi_status, Buffer *recv_buff, MPI_Request &recv_2nd_req, MPI_Request &ack_req, int &this_incoming, Buffer *send_buff, MPI_Request &send_req, MPI_Request &sent_ack_req, bool &done, Buffer *next_buff=NULL, int next_tag=-1, MPI_Request *next_req=NULL, int *next_incoming=NULL)
 process incoming message; if longer than the initial size, post recv for next part then send ack; if ack, send second part; else indicate that we're done and buffer is ready for processing More...
 
ErrorCode pack_entity_seq (const int nodes_per_entity, const bool store_remote_handles, const int to_proc, Range &these_ents, std::vector< EntityHandle > &entities, Buffer *buff)
 pack a range of entities with equal # verts per entity, along with the range on the sending proc More...
 
ErrorCode print_buffer (unsigned char *buff_ptr, int mesg_type, int from_proc, bool sent)
 
ErrorCode unpack_iface_entities (unsigned char *&buff_ptr, const int from_proc, const int ind, std::vector< EntityHandle > &recd_ents)
 for all the entities in the received buffer; for each, save entities in this instance which match connectivity, or zero if none found More...
 
ErrorCode pack_sets (Range &entities, Buffer *buff, const bool store_handles, const int to_proc)
 
ErrorCode unpack_sets (unsigned char *&buff_ptr, std::vector< EntityHandle > &entities, const bool store_handles, const int to_proc)
 
ErrorCode pack_adjacencies (Range &entities, Range::const_iterator &start_rit, Range &whole_range, unsigned char *&buff_ptr, int &count, const bool just_count, const bool store_handles, const int to_proc)
 
ErrorCode unpack_adjacencies (unsigned char *&buff_ptr, Range &entities, const bool store_handles, const int from_proc)
 
ErrorCode unpack_remote_handles (unsigned int from_proc, const unsigned char *buff_ptr, std::vector< EntityHandle > &L2hloc, std::vector< EntityHandle > &L2hrem, std::vector< unsigned int > &L2p)
 
ErrorCode find_existing_entity (const bool is_iface, const int owner_p, const EntityHandle owner_h, const int num_ents, const EntityHandle *connect, const int num_connect, const EntityType this_type, std::vector< EntityHandle > &L2hloc, std::vector< EntityHandle > &L2hrem, std::vector< unsigned int > &L2p, EntityHandle &new_h)
 given connectivity and type, find an existing entity, if there is one More...
 
ErrorCode build_sharedhps_list (const EntityHandle entity, const unsigned char pstatus, const int sharedp, const std::set< unsigned int > &procs, unsigned int &num_ents, int *tmp_procs, EntityHandle *tmp_handles)
 
ErrorCode get_tag_send_list (const Range &all_entities, std::vector< Tag > &all_tags, std::vector< Range > &tag_ranges)
 Get list of tags for which to exchange data. More...
 
ErrorCode pack_tags (Range &entities, const std::vector< Tag > &src_tags, const std::vector< Tag > &dst_tags, const std::vector< Range > &tag_ranges, Buffer *buff, const bool store_handles, const int to_proc)
 Serialize entity tag data. More...
 
ErrorCode packed_tag_size (Tag source_tag, const Range &entities, int &count_out)
 Calculate buffer size required to pack tag data. More...
 
ErrorCode pack_tag (Tag source_tag, Tag destination_tag, const Range &entities, const std::vector< EntityHandle > &whole_range, Buffer *buff, const bool store_remote_handles, const int to_proc)
 Serialize tag data. More...
 
ErrorCode unpack_tags (unsigned char *&buff_ptr, std::vector< EntityHandle > &entities, const bool store_handles, const int to_proc, const MPI_Op *const mpi_op=NULL)
 
ErrorCode tag_shared_verts (TupleList &shared_verts, Range *skin_ents, std::map< std::vector< int >, std::vector< EntityHandle > > &proc_nvecs, Range &proc_verts)
 
ErrorCode get_proc_nvecs (int resolve_dim, int shared_dim, Range *skin_ents, std::map< std::vector< int >, std::vector< EntityHandle > > &proc_nvecs)
 
ErrorCode create_iface_pc_links ()
 
ErrorCode pack_range_map (Range &this_range, EntityHandle actual_start, HandleMap &handle_map)
 pack a range map with keys in this_range and values a contiguous series of handles starting at actual_start More...
 
bool is_iface_proc (EntityHandle this_set, int to_proc)
 returns true if the set is an interface shared with to_proc More...
 
ErrorCode update_iface_sets (Range &sent_ents, std::vector< EntityHandle > &remote_handles, int from_proc)
 for any remote_handles set to zero, remove corresponding sent_ents from iface_sets corresponding to from_proc More...
 
ErrorCode get_ghosted_entities (int bridge_dim, int ghost_dim, int to_proc, int num_layers, int addl_ents, Range &ghosted_ents)
 for specified bridge/ghost dimension, to_proc, and number of layers, get the entities to be ghosted, and info on additional procs needing to communicate with to_proc More...
 
ErrorCode add_verts (Range &sent_ents)
 add vertices adjacent to entities in this list More...
 
ErrorCode exchange_all_shared_handles (std::vector< std::vector< SharedEntityData > > &send_data, std::vector< std::vector< SharedEntityData > > &result)
 Every processor sends shared entity handle data to every other processor that it shares entities with. Passed back map is all received data, indexed by processor ID. This function is intended to be used for debugging. More...
 
ErrorCode get_remote_handles (const bool store_remote_handles, EntityHandle *from_vec, EntityHandle *to_vec_tmp, int num_ents, int to_proc, const std::vector< EntityHandle > &new_ents)
 replace handles in from_vec with corresponding handles on to_proc (by checking shared[p/h]_tag and shared[p/h]s_tag; if no remote handle and new_ents is non-null, substitute instead CREATE_HANDLE(MBMAXTYPE, index) where index is handle's position in new_ents More...
 
ErrorCode get_remote_handles (const bool store_remote_handles, const Range &from_range, Range &to_range, int to_proc, const std::vector< EntityHandle > &new_ents)
 same as other version, except from_range and to_range should be different here More...
 
ErrorCode get_remote_handles (const bool store_remote_handles, const Range &from_range, EntityHandle *to_vec, int to_proc, const std::vector< EntityHandle > &new_ents)
 same as other version, except packs range into vector More...
 
ErrorCode get_local_handles (EntityHandle *from_vec, int num_ents, const Range &new_ents)
 goes through from_vec, and for any with type MBMAXTYPE, replaces with new_ents value at index corresponding to id of entity in from_vec More...
 
ErrorCode get_local_handles (const Range &remote_handles, Range &local_handles, const std::vector< EntityHandle > &new_ents)
 same as above except puts results in range More...
 
ErrorCode get_local_handles (EntityHandle *from_vec, int num_ents, const std::vector< EntityHandle > &new_ents)
 same as above except gets new_ents from vector More...
 
ErrorCode update_remote_data (Range &local_range, Range &remote_range, int other_proc, const unsigned char add_pstat)
 
ErrorCode update_remote_data (const EntityHandle new_h, const int *ps, const EntityHandle *hs, const int num_ps, const unsigned char add_pstat)
 
ErrorCode update_remote_data_old (const EntityHandle new_h, const int *ps, const EntityHandle *hs, const int num_ps, const unsigned char add_pstat)
 
ErrorCode tag_iface_entities ()
 Set pstatus tag interface bit on entities in sets passed in. More...
 
int add_pcomm (ParallelComm *pc)
 add a pc to the iface instance tag PARALLEL_COMM More...
 
void remove_pcomm (ParallelComm *pc)
 remove a pc from the iface instance tag PARALLEL_COMM More...
 
ErrorCode check_sent_ents (Range &allsent)
 check entities to make sure there are no zero-valued remote handles where they shouldn't be More...
 
ErrorCode assign_entities_part (std::vector< EntityHandle > &entities, const int proc)
 assign entities to the input processor part More...
 
ErrorCode remove_entities_part (Range &entities, const int proc)
 remove entities to the input processor part More...
 
void delete_all_buffers ()
 reset message buffers to their initial state More...
 

Private Attributes

Interface * mbImpl
 MB interface associated with this writer. More...
 
ProcConfig procConfig
 Proc config object, keeps info on parallel stuff. More...
 
SequenceManager * sequenceManager
 Sequence manager, to get more efficient access to entities. More...
 
Error * errorHandler
 Error handler. More...
 
std::vector< Buffer * > localOwnedBuffs
 more data buffers, proc-specific More...
 
std::vector< Buffer * > remoteOwnedBuffs
 
std::vector< MPI_Request > sendReqs
 request objects, may be used if store_remote_handles is used More...
 
std::vector< MPI_Request > recvReqs
 receive request objects More...
 
std::vector< MPI_Request > recvRemotehReqs
 
std::vector< unsigned int > buffProcs
 processor rank for each buffer index More...
 
Range partitionSets
 the partition, interface sets for this comm'n instance More...
 
Range interfaceSets
 
std::set< EntityHandle > sharedEnts
 all local entities shared with others, whether ghost or ghosted More...
 
Tag sharedpTag
 tags used to save sharing procs and handles More...
 
Tag sharedpsTag
 
Tag sharedhTag
 
Tag sharedhsTag
 
Tag pstatusTag
 
Tag ifaceSetsTag
 
Tag partitionTag
 
int globalPartCount
 Cache of global part count. More...
 
EntityHandle partitioningSet
 entity set containing all parts More...
 
std::ofstream myFile
 
int pcommID
 
int ackbuff
 
DebugOutput * myDebug
 used to set verbosity level and to report output More...
 
SharedSetData * sharedSetData
 Data about shared sets. More...
 

Friends

class ParallelMergeMesh
 

Detailed Description

Parallel communications in MOAB.

Author
Tim Tautges

This class implements methods to communicate mesh between processors

Examples
ComputeTriDual.cpp, ExchangeHalos.cpp, HelloParMOAB.cpp, and ReadWriteTest.cpp.

Definition at line 54 of file ParallelComm.hpp.

Constructor & Destructor Documentation

◆ ParallelComm() [1/2]

moab::ParallelComm::ParallelComm ( Interface *  impl,
MPI_Comm  comm,
int *  pcomm_id_out = 0 
)

constructor

Definition at line 313 of file ParallelComm.cpp.

314  : mbImpl( impl ), procConfig( cm ), sharedpTag( 0 ), sharedpsTag( 0 ), sharedhTag( 0 ), sharedhsTag( 0 ),
315  pstatusTag( 0 ), ifaceSetsTag( 0 ), partitionTag( 0 ), globalPartCount( -1 ), partitioningSet( 0 ),
316  myDebug( NULL )
317 {
318  initialize();
319  sharedSetData = new SharedSetData( *impl, pcommID, procConfig.proc_rank() );
320  if( id ) *id = pcommID;
321 }

References initialize(), pcommID, moab::ProcConfig::proc_rank(), procConfig, and sharedSetData.

Referenced by get_pcomm().

◆ ParallelComm() [2/2]

moab::ParallelComm::ParallelComm ( Interface *  impl,
std::vector< unsigned char > &  tmp_buff,
MPI_Comm  comm,
int *  pcomm_id_out = 0 
)

constructor taking packed buffer, for testing

Definition at line 323 of file ParallelComm.cpp.

324  : mbImpl( impl ), procConfig( cm ), sharedpTag( 0 ), sharedpsTag( 0 ), sharedhTag( 0 ), sharedhsTag( 0 ),
325  pstatusTag( 0 ), ifaceSetsTag( 0 ), partitionTag( 0 ), globalPartCount( -1 ), partitioningSet( 0 ),
326  myDebug( NULL )
327 {
328  initialize();
329  sharedSetData = new SharedSetData( *impl, pcommID, procConfig.proc_rank() );
330  if( id ) *id = pcommID;
331 }

References initialize(), pcommID, moab::ProcConfig::proc_rank(), procConfig, and sharedSetData.

◆ ~ParallelComm()

moab::ParallelComm::~ParallelComm ( )

destructor

Definition at line 333 of file ParallelComm.cpp.

334 {
335  remove_pcomm( this );
336  delete_all_buffers();
337  delete myDebug;
338  delete sharedSetData;
339 }

References delete_all_buffers(), myDebug, remove_pcomm(), and sharedSetData.

Member Function Documentation

◆ add_pcomm()

int moab::ParallelComm::add_pcomm ( ParallelComm *  pc)
private

add a pc to the iface instance tag PARALLEL_COMM

Definition at line 374 of file ParallelComm.cpp.

375 {
376  // Add this pcomm to instance tag
377  std::vector< ParallelComm* > pc_array( MAX_SHARING_PROCS, (ParallelComm*)NULL );
378  Tag pc_tag = pcomm_tag( mbImpl, true );
379  assert( 0 != pc_tag );
380 
381  const EntityHandle root = 0;
382  ErrorCode result = mbImpl->tag_get_data( pc_tag, &root, 1, (void*)pc_array.data() );
383  if( MB_SUCCESS != result && MB_TAG_NOT_FOUND != result ) return -1;
384  int index = 0;
385  while( index < MAX_SHARING_PROCS && pc_array[index] )
386  index++;
387  if( index == MAX_SHARING_PROCS )
388  {
389  index = -1;
390  assert( false );
391  }
392  else
393  {
394  pc_array[index] = pc;
395  mbImpl->tag_set_data( pc_tag, &root, 1, (void*)pc_array.data() );
396  }
397  return index;
398 }

References ErrorCode, moab::index, MAX_SHARING_PROCS, MB_SUCCESS, MB_TAG_NOT_FOUND, mbImpl, pcomm_tag(), moab::Interface::tag_get_data(), and moab::Interface::tag_set_data().

Referenced by initialize().

◆ add_verts()

ErrorCode moab::ParallelComm::add_verts ( Range &  sent_ents)
private

add vertices adjacent to entities in this list

Definition at line 7794 of file ParallelComm.cpp.

7795 {
7796  // Get the verts adj to these entities, since we'll have to send those too
7797 
7798  // First check sets
7799  std::pair< Range::const_iterator, Range::const_iterator > set_range = sent_ents.equal_range( MBENTITYSET );
7800  ErrorCode result = MB_SUCCESS, tmp_result;
7801  for( Range::const_iterator rit = set_range.first; rit != set_range.second; ++rit )
7802  {
7803  tmp_result = mbImpl->get_entities_by_type( *rit, MBVERTEX, sent_ents );
7804  MB_CHK_SET_ERR( tmp_result, "Failed to get contained verts" );
7805  }
7806 
7807  // Now non-sets
7808  Range tmp_ents;
7809  std::copy( sent_ents.begin(), set_range.first, range_inserter( tmp_ents ) );
7810  result = mbImpl->get_adjacencies( tmp_ents, 0, false, sent_ents, Interface::UNION );
7811  MB_CHK_SET_ERR( result, "Failed to get vertices adj to ghosted ents" );
7812 
7813  // if polyhedra, need to add all faces from there
7814  Range polyhedra = sent_ents.subset_by_type( MBPOLYHEDRON );
7815  // get all faces adjacent to every polyhedra
7816  result = mbImpl->get_connectivity( polyhedra, sent_ents );
7817  MB_CHK_SET_ERR( result, "Failed to get polyhedra faces" );
7818  return result;
7819 }

References moab::Range::begin(), moab::Range::equal_range(), ErrorCode, moab::Interface::get_adjacencies(), moab::Interface::get_connectivity(), moab::Interface::get_entities_by_type(), MB_CHK_SET_ERR, MB_SUCCESS, MBENTITYSET, mbImpl, MBPOLYHEDRON, MBVERTEX, moab::Range::subset_by_type(), and moab::Interface::UNION.

Referenced by broadcast_entities(), exchange_owned_mesh(), get_ghosted_entities(), scatter_entities(), and send_entities().

◆ assign_entities_part()

ErrorCode moab::ParallelComm::assign_entities_part ( std::vector< EntityHandle > &  entities,
const int  proc 
)
private

assign entities to the input processor part

Definition at line 7570 of file ParallelComm.cpp.

7571 {
7572  EntityHandle part_set;
7573  ErrorCode result = get_part_handle( proc, part_set );
7574  MB_CHK_SET_ERR( result, "Failed to get part handle" );
7575 
7576  if( part_set > 0 )
7577  {
7578  result = mbImpl->add_entities( part_set, entities.data(), entities.size() );
7579  MB_CHK_SET_ERR( result, "Failed to add entities to part set" );
7580  }
7581 
7582  return MB_SUCCESS;
7583 }

References moab::Interface::add_entities(), ErrorCode, get_part_handle(), MB_CHK_SET_ERR, MB_SUCCESS, and mbImpl.

Referenced by exchange_owned_mesh(), and recv_entities().

◆ assign_global_ids() [1/2]

ErrorCode moab::ParallelComm::assign_global_ids ( EntityHandle  this_set,
const int  dimension,
const int  start_id = 1,
const bool  largest_dim_only = true,
const bool  parallel = true,
const bool  owned_only = false 
)

assign a global id space, for largest-dimension or all entities (and in either case for vertices too)

Assign a global id space, for largest-dimension or all entities (and in either case for vertices too)

Parameters
owned_onlyIf true, do not get global IDs for non-owned entities from remote processors.
Examples
ComputeTriDual.cpp.

Definition at line 421 of file ParallelComm.cpp.

427 {
428  Range entities[4];
429  ErrorCode result;
430  std::vector< unsigned char > pstatus;
431  for( int dim = 0; dim <= dimension; dim++ )
432  {
433  if( dim == 0 || !largest_dim_only || dim == dimension )
434  {
435  result = mbImpl->get_entities_by_dimension( this_set, dim, entities[dim] );
436  MB_CHK_SET_ERR( result, "Failed to get vertices in assign_global_ids" );
437  }
438 
439  // Need to filter out non-locally-owned entities!!!
440  pstatus.resize( entities[dim].size() );
441  result = mbImpl->tag_get_data( pstatus_tag(), entities[dim], pstatus.data() );
442  MB_CHK_SET_ERR( result, "Failed to get pstatus in assign_global_ids" );
443 
444  Range dum_range;
445  Range::iterator rit;
446  unsigned int i;
447  for( rit = entities[dim].begin(), i = 0; rit != entities[dim].end(); ++rit, i++ )
448  if( pstatus[i] & PSTATUS_NOT_OWNED ) dum_range.insert( *rit );
449  entities[dim] = subtract( entities[dim], dum_range );
450  }
451 
452  return assign_global_ids( entities, dimension, start_id, parallel, owned_only );
453 }

References moab::Range::end(), ErrorCode, moab::Interface::get_entities_by_dimension(), moab::Range::insert(), MB_CHK_SET_ERR, mbImpl, PSTATUS_NOT_OWNED, pstatus_tag(), size(), moab::subtract(), and moab::Interface::tag_get_data().

Referenced by check_global_ids(), compute_dual_mesh(), moab::NCHelperDomain::create_mesh(), moab::NCHelperScrip::create_mesh(), iMOAB_MergeVertices(), main(), and resolve_shared_ents().

◆ assign_global_ids() [2/2]

ErrorCode moab::ParallelComm::assign_global_ids ( Range  entities[],
const int  dimension,
const int  start_id,
const bool  parallel,
const bool  owned_only 
)

assign a global id space, for largest-dimension or all entities (and in either case for vertices too)

Assign a global id space, for largest-dimension or all entities (and in either case for vertices too)

Definition at line 457 of file ParallelComm.cpp.

462 {
463  int local_num_elements[4];
464  ErrorCode result;
465  for( int dim = 0; dim <= dimension; dim++ )
466  {
467  local_num_elements[dim] = entities[dim].size();
468  }
469 
470  // Communicate numbers
471  std::vector< int > num_elements( procConfig.proc_size() * 4 );
472 #ifdef MOAB_HAVE_MPI
473  if( procConfig.proc_size() > 1 && parallel )
474  {
475  int retval =
476  MPI_Allgather( local_num_elements, 4, MPI_INT, num_elements.data(), 4, MPI_INT, procConfig.proc_comm() );
477  if( 0 != retval ) return MB_FAILURE;
478  }
479  else
480 #endif
481  for( int dim = 0; dim < 4; dim++ )
482  num_elements[dim] = local_num_elements[dim];
483 
484  // My entities start at one greater than total_elems[d]
485  int total_elems[4] = { start_id, start_id, start_id, start_id };
486 
487  for( unsigned int proc = 0; proc < procConfig.proc_rank(); proc++ )
488  {
489  for( int dim = 0; dim < 4; dim++ )
490  total_elems[dim] += num_elements[4 * proc + dim];
491  }
492 
493  // Assign global ids now
494  Tag gid_tag = mbImpl->globalId_tag();
495 
496  for( int dim = 0; dim < 4; dim++ )
497  {
498  if( entities[dim].empty() ) continue;
499  num_elements.resize( entities[dim].size() );
500  int i = 0;
501  for( Range::iterator rit = entities[dim].begin(); rit != entities[dim].end(); ++rit )
502  num_elements[i++] = total_elems[dim]++;
503 
504  result = mbImpl->tag_set_data( gid_tag, entities[dim], num_elements.data() );
505  MB_CHK_SET_ERR( result, "Failed to set global id tag in assign_global_ids" );
506  }
507 
508  if( owned_only ) return MB_SUCCESS;
509 
510  // Exchange tags
511  for( int dim = 1; dim < 4; dim++ )
512  entities[0].merge( entities[dim] );
513 
514  return exchange_tags( gid_tag, entities[0] );
515 }

References moab::Range::end(), ErrorCode, exchange_tags(), moab::Interface::globalId_tag(), MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_rank(), moab::ProcConfig::proc_size(), procConfig, moab::Range::size(), size(), and moab::Interface::tag_set_data().

◆ augment_default_sets_with_ghosts()

ErrorCode moab::ParallelComm::augment_default_sets_with_ghosts ( EntityHandle  file_set)

extend shared sets with ghost entities After ghosting, ghost entities do not have yet information about the material set, partition set, Neumann or Dirichlet set they could belong to This method will assign ghosted entities to the those special entity sets In some case we might even have to create those sets, if they do not exist yet on the local processor

The special entity sets all have an unique identifier, in a form of an integer tag to the set. The shared sets data is not used, because we do not use the geometry sets, as they are not uniquely identified

Parameters
file_set: file set used per application

Definition at line 4922 of file ParallelComm.cpp.

4923 {
4924  // gather all default sets we are interested in, material, neumann, etc
4925  // we will skip geometry sets, because they are not uniquely identified with their tag value
4926  // maybe we will add another tag, like category
4927 
4928  if( procConfig.proc_size() < 2 ) return MB_SUCCESS; // no reason to stop by
4929  const char* const shared_set_tag_names[] = { MATERIAL_SET_TAG_NAME, DIRICHLET_SET_TAG_NAME, NEUMANN_SET_TAG_NAME,
4930  PARALLEL_PARTITION_TAG_NAME };
4931 
4932  int num_tags = sizeof( shared_set_tag_names ) / sizeof( shared_set_tag_names[0] );
4933 
4934  Range* rangeSets = new Range[num_tags];
4935  Tag* tags = new Tag[num_tags + 1]; // one extra for global id tag, which is an int, so far
4936 
4937  int my_rank = rank();
4938  int** tagVals = new int*[num_tags];
4939  for( int i = 0; i < num_tags; i++ )
4940  tagVals[i] = NULL;
4941  ErrorCode rval;
4942 
4943  // for each tag, we keep a local map, from the value to the actual set with that value
4944  // we assume that the tag values are unique, for a given set, otherwise we
4945  // do not know to which set to add the entity
4946 
4947  typedef std::map< int, EntityHandle > MVal;
4948  typedef std::map< int, EntityHandle >::iterator itMVal;
4949  MVal* localMaps = new MVal[num_tags];
4950 
4951  for( int i = 0; i < num_tags; i++ )
4952  {
4953 
4954  rval = mbImpl->tag_get_handle( shared_set_tag_names[i], 1, MB_TYPE_INTEGER, tags[i], MB_TAG_ANY );
4955  if( MB_SUCCESS != rval ) continue;
4956  MB_CHK_SET_ERR( mbImpl->get_entities_by_type_and_tag( file_set, MBENTITYSET, &( tags[i] ), 0, 1, rangeSets[i],
4957  Interface::UNION ),
4958  "can't get sets with a tag" );
4959 
4960  if( rangeSets[i].size() > 0 )
4961  {
4962  tagVals[i] = new int[rangeSets[i].size()];
4963  // fill up with the tag values
4964  MB_CHK_SET_ERR( mbImpl->tag_get_data( tags[i], rangeSets[i], tagVals[i] ), "can't get set tag values" );
4965  // now for inverse mapping:
4966  for( int j = 0; j < (int)rangeSets[i].size(); j++ )
4967  {
4968  localMaps[i][tagVals[i][j]] = rangeSets[i][j];
4969  }
4970  }
4971  }
4972  // get the global id tag too
4973  tags[num_tags] = mbImpl->globalId_tag();
4974 
4975  TupleList remoteEnts;
4976  // processor to send to, type of tag (0-mat,) tag value, remote handle
4977  // 1-diri
4978  // 2-neum
4979  // 3-part
4980  //
4981  int initialSize = (int)sharedEnts.size(); // estimate that on average, each shared ent
4982  // will be sent to one processor, for one tag
4983  // we will actually send only entities that are owned locally, and from those
4984  // only those that do have a special tag (material, neumann, etc)
4985  // if we exceed the capacity, we resize the tuple
4986  remoteEnts.initialize( 3, 0, 1, 0, initialSize );
4987  remoteEnts.enableWriteAccess();
4988 
4989  // now, for each owned entity, get the remote handle(s) and Proc(s), and verify if it
4990  // belongs to one of the sets; if yes, create a tuple and append it
4991 
4992  std::set< EntityHandle > own_and_sha;
4993  int ir = 0, jr = 0;
4994  for( std::set< EntityHandle >::iterator vit = sharedEnts.begin(); vit != sharedEnts.end(); ++vit )
4995  {
4996  // ghosted eh
4997  EntityHandle geh = *vit;
4998  if( own_and_sha.find( geh ) != own_and_sha.end() ) // already encountered
4999  continue;
5000  int procs[MAX_SHARING_PROCS];
5001  EntityHandle handles[MAX_SHARING_PROCS];
5002  int nprocs;
5003  unsigned char pstat;
5004  rval = get_sharing_data( geh, procs, handles, pstat, nprocs );
5005  if( rval != MB_SUCCESS )
5006  {
5007  for( int i = 0; i < num_tags; i++ )
5008  delete[] tagVals[i];
5009  delete[] tagVals;MB_CHK_SET_ERR( rval, "Failed to get sharing data" );
5010  }
5011  if( pstat & PSTATUS_NOT_OWNED ) continue; // we will send info only for entities that we own
5012  own_and_sha.insert( geh );
5013  for( int i = 0; i < num_tags; i++ )
5014  {
5015  for( int j = 0; j < (int)rangeSets[i].size(); j++ )
5016  {
5017  EntityHandle specialSet = rangeSets[i][j]; // this set has tag i, value tagVals[i][j];
5018  if( mbImpl->contains_entities( specialSet, &geh, 1 ) )
5019  {
5020  // this ghosted entity is in a special set, so form the tuple
5021  // to send to the processors that do not own this
5022  for( int k = 0; k < nprocs; k++ )
5023  {
5024  if( procs[k] != my_rank )
5025  {
5026  if( remoteEnts.get_n() >= remoteEnts.get_max() - 1 )
5027  {
5028  // resize, so we do not overflow
5029  int oldSize = remoteEnts.get_max();
5030  // increase with 50% the capacity
5031  remoteEnts.resize( oldSize + oldSize / 2 + 1 );
5032  }
5033  remoteEnts.vi_wr[ir++] = procs[k]; // send to proc
5034  remoteEnts.vi_wr[ir++] = i; // for the tags [i] (0-3)
5035  remoteEnts.vi_wr[ir++] = tagVals[i][j]; // actual value of the tag
5036  remoteEnts.vul_wr[jr++] = handles[k];
5037  remoteEnts.inc_n();
5038  }
5039  }
5040  }
5041  }
5042  }
5043  // if the local entity has a global id, send it too, so we avoid
5044  // another "exchange_tags" for global id
5045  int gid;
5046  MB_CHK_SET_ERR( mbImpl->tag_get_data( tags[num_tags], &geh, 1, &gid ), "Failed to get global id" );
5047  if( gid != 0 )
5048  {
5049  for( int k = 0; k < nprocs; k++ )
5050  {
5051  if( procs[k] != my_rank )
5052  {
5053  if( remoteEnts.get_n() >= remoteEnts.get_max() - 1 )
5054  {
5055  // resize, so we do not overflow
5056  int oldSize = remoteEnts.get_max();
5057  // increase with 50% the capacity
5058  remoteEnts.resize( oldSize + oldSize / 2 + 1 );
5059  }
5060  remoteEnts.vi_wr[ir++] = procs[k]; // send to proc
5061  remoteEnts.vi_wr[ir++] = num_tags; // for the tags [j] (4)
5062  remoteEnts.vi_wr[ir++] = gid; // actual value of the tag
5063  remoteEnts.vul_wr[jr++] = handles[k];
5064  remoteEnts.inc_n();
5065  }
5066  }
5067  }
5068  }
5069 
5070 #ifndef NDEBUG
5071  if( my_rank == 1 && 1 == get_debug_verbosity() ) remoteEnts.print( " on rank 1, before augment routing" );
5072  MPI_Barrier( procConfig.proc_comm() );
5073  int sentEnts = remoteEnts.get_n();
5074  assert( ( sentEnts == jr ) && ( 3 * sentEnts == ir ) );
5075 #endif
5076  // exchange the info now, and send to
5077  gs_data::crystal_data* cd = this->procConfig.crystal_router();
5078  // All communication happens here; no other mpi calls
5079  // Also, this is a collective call
5080  MB_CHK_SET_ERR( cd->gs_transfer( 1, remoteEnts, 0 ), "Error in tuple transfer" );
5081 #ifndef NDEBUG
5082  if( my_rank == 0 && 1 == get_debug_verbosity() ) remoteEnts.print( " on rank 0, after augment routing" );
5083  MPI_Barrier( procConfig.proc_comm() );
5084 #endif
5085 
5086  // now process the data received from other processors
5087  int received = remoteEnts.get_n();
5088  for( int i = 0; i < received; i++ )
5089  {
5090  // int from = ents_to_delete.vi_rd[i];
5091  EntityHandle geh = (EntityHandle)remoteEnts.vul_rd[i];
5092  int from_proc = remoteEnts.vi_rd[3 * i];
5093  if( my_rank == from_proc )
5094  std::cout << " unexpected receive from my rank " << my_rank << " during augmenting with ghosts\n ";
5095  int tag_type = remoteEnts.vi_rd[3 * i + 1];
5096  assert( ( 0 <= tag_type ) && ( tag_type <= num_tags ) );
5097  int value = remoteEnts.vi_rd[3 * i + 2];
5098  if( tag_type == num_tags )
5099  {
5100  // it is global id
5101  MB_CHK_SET_ERR( mbImpl->tag_set_data( tags[num_tags], &geh, 1, &value ), "Error in setting gid tag" );
5102  }
5103  else
5104  {
5105  // now, based on value and tag type, see if we have that value in the map
5106  MVal& lmap = localMaps[tag_type];
5107  itMVal itm = lmap.find( value );
5108  if( itm == lmap.end() )
5109  {
5110  // the value was not found yet in the local map, so we have to create the set
5111  EntityHandle newSet;
5112  MB_CHK_SET_ERR( mbImpl->create_meshset( MESHSET_SET, newSet ), "can't create new set" );
5113  lmap[value] = newSet;
5114  // set the tag value
5115  MB_CHK_SET_ERR( mbImpl->tag_set_data( tags[tag_type], &newSet, 1, &value ),
5116  "can't set tag for new set" );
5117 
5118  // we also need to add the new created set to the file set, if not null
5119  if( file_set )
5120  {
5121  MB_CHK_SET_ERR( mbImpl->add_entities( file_set, &newSet, 1 ), "can't add new set to the file set" );
5122  }
5123  }
5124  // add the entity to the set pointed to by the map
5125  MB_CHK_SET_ERR( mbImpl->add_entities( lmap[value], &geh, 1 ), "can't add ghost ent to the set" );
5126  }
5127  }
5128 
5129  for( int i = 0; i < num_tags; i++ )
5130  delete[] tagVals[i];
5131  delete[] tagVals;
5132  delete[] rangeSets;
5133  delete[] tags;
5134  delete[] localMaps;
5135  return MB_SUCCESS;
5136 }

References moab::Interface::add_entities(), moab::Interface::contains_entities(), moab::Interface::create_meshset(), moab::ProcConfig::crystal_router(), DIRICHLET_SET_TAG_NAME, moab::TupleList::enableWriteAccess(), ErrorCode, get_debug_verbosity(), moab::Interface::get_entities_by_type_and_tag(), moab::TupleList::get_max(), moab::TupleList::get_n(), get_sharing_data(), moab::Interface::globalId_tag(), moab::TupleList::inc_n(), moab::TupleList::initialize(), MATERIAL_SET_TAG_NAME, MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SUCCESS, MB_TAG_ANY, MB_TYPE_INTEGER, MBENTITYSET, mbImpl, MESHSET_SET, NEUMANN_SET_TAG_NAME, PARALLEL_PARTITION_TAG_NAME, moab::TupleList::print(), moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_size(), procConfig, PSTATUS_NOT_OWNED, rank(), moab::TupleList::resize(), sharedEnts, moab::Range::size(), size(), moab::Interface::tag_get_data(), moab::Interface::tag_get_handle(), moab::Interface::tag_set_data(), moab::Interface::UNION, moab::TupleList::vi_rd, moab::TupleList::vi_wr, moab::TupleList::vul_rd, and moab::TupleList::vul_wr.

Referenced by moab::ReadParallel::load_file().

◆ broadcast_entities()

ErrorCode moab::ParallelComm::broadcast_entities ( const int  from_proc,
Range &  entities,
const bool  adjacencies = false,
const bool  tags = true 
)

Broadcast all entities resident on from_proc to other processors This function assumes remote handles are not being stored, since (usually) every processor will know about the whole mesh.

Parameters
from_procProcessor having the mesh to be broadcast
entitiesOn return, the entities sent or received in this call
adjacenciesIf true, adjacencies are sent for equiv entities (currently unsupported)
tagsIf true, all non-default-valued tags are sent for sent entities

Definition at line 539 of file ParallelComm.cpp.

543 {
544 #ifndef MOAB_HAVE_MPI
545  return MB_FAILURE;
546 #else
547 
548  ErrorCode result = MB_SUCCESS;
549  int success;
550  int buff_size;
551 
552  Buffer buff( INITIAL_BUFF_SIZE );
553  buff.reset_ptr( sizeof( int ) );
554  if( (int)procConfig.proc_rank() == from_proc )
555  {
556  result = add_verts( entities );
557  MB_CHK_SET_ERR( result, "Failed to add adj vertices" );
558 
559  buff.reset_ptr( sizeof( int ) );
560  result = pack_buffer( entities, adjacencies, tags, false, -1, &buff );
561  MB_CHK_SET_ERR( result, "Failed to compute buffer size in broadcast_entities" );
562  buff.set_stored_size();
563  buff_size = buff.buff_ptr - buff.mem_ptr;
564  }
565 
566  success = MPI_Bcast( &buff_size, 1, MPI_INT, from_proc, procConfig.proc_comm() );
567  if( MPI_SUCCESS != success )
568  {
569  MB_SET_ERR( MB_FAILURE, "MPI_Bcast of buffer size failed" );
570  }
571 
572  if( !buff_size ) // No data
573  return MB_SUCCESS;
574 
575  if( (int)procConfig.proc_rank() != from_proc ) buff.reserve( buff_size );
576 
577  size_t offset = 0;
578  while( buff_size )
579  {
580  int sz = std::min( buff_size, MAX_BCAST_SIZE );
581  success = MPI_Bcast( buff.mem_ptr + offset, sz, MPI_UNSIGNED_CHAR, from_proc, procConfig.proc_comm() );
582  if( MPI_SUCCESS != success )
583  {
584  MB_SET_ERR( MB_FAILURE, "MPI_Bcast of buffer failed" );
585  }
586 
587  offset += sz;
588  buff_size -= sz;
589  }
590 
591  if( (int)procConfig.proc_rank() != from_proc )
592  {
593  std::vector< std::vector< EntityHandle > > dum1a, dum1b;
594  std::vector< std::vector< int > > dum1p;
595  std::vector< EntityHandle > dum2, dum4;
596  std::vector< unsigned int > dum3;
597  buff.reset_ptr( sizeof( int ) );
598  result = unpack_buffer( buff.buff_ptr, false, from_proc, -1, dum1a, dum1b, dum1p, dum2, dum2, dum3, dum4 );
599  MB_CHK_SET_ERR( result, "Failed to unpack buffer in broadcast_entities" );
600  std::copy( dum4.begin(), dum4.end(), range_inserter( entities ) );
601  }
602 
603  return MB_SUCCESS;
604 #endif
605 }

References add_verts(), moab::ParallelComm::Buffer::buff_ptr, ErrorCode, INITIAL_BUFF_SIZE, moab::MAX_BCAST_SIZE, MB_CHK_SET_ERR, MB_SET_ERR, MB_SUCCESS, moab::ParallelComm::Buffer::mem_ptr, pack_buffer(), moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_rank(), procConfig, moab::ParallelComm::Buffer::reserve(), moab::ParallelComm::Buffer::reset_ptr(), moab::ParallelComm::Buffer::set_stored_size(), and unpack_buffer().

Referenced by moab::ReadParallel::load_file().

◆ buff_procs()

const std::vector< unsigned int > & moab::ParallelComm::buff_procs ( ) const
inline

get buff processor vector

Definition at line 1569 of file ParallelComm.hpp.

1570 {
1571  return buffProcs;
1572 }

References buffProcs.

◆ build_sharedhps_list()

ErrorCode moab::ParallelComm::build_sharedhps_list ( const EntityHandle  entity,
const unsigned char  pstatus,
const int  sharedp,
const std::set< unsigned int > &  procs,
unsigned int &  num_ents,
int *  tmp_procs,
EntityHandle *  tmp_handles 
)
private

Definition at line 1797 of file ParallelComm.cpp.

1808 {
1809  num_ents = 0;
1810  unsigned char pstat;
1811  ErrorCode result = get_sharing_data( entity, tmp_procs, tmp_handles, pstat, num_ents );
1812  MB_CHK_SET_ERR( result, "Failed to get sharing data" );
1813  assert( pstat == pstatus );
1814 
1815  // Build shared proc/handle lists
1816  // Start with multi-shared, since if it is the owner will be first
1817  if( pstatus & PSTATUS_MULTISHARED )
1818  {
1819  }
1820  else if( pstatus & PSTATUS_NOT_OWNED )
1821  {
1822  // If not multishared and not owned, other sharing proc is owner, put that
1823  // one first
1824  assert( "If not owned, I should be shared too" && pstatus & PSTATUS_SHARED && 1 == num_ents );
1825  tmp_procs[1] = procConfig.proc_rank();
1826  tmp_handles[1] = entity;
1827  num_ents = 2;
1828  }
1829  else if( pstatus & PSTATUS_SHARED )
1830  {
1831  // If not multishared and owned, I'm owner
1832  assert( "shared and owned, should be only 1 sharing proc" && 1 == num_ents );
1833  tmp_procs[1] = tmp_procs[0];
1834  tmp_procs[0] = procConfig.proc_rank();
1835  tmp_handles[1] = tmp_handles[0];
1836  tmp_handles[0] = entity;
1837  num_ents = 2;
1838  }
1839  else
1840  {
1841  // Not shared yet, just add owner (me)
1842  tmp_procs[0] = procConfig.proc_rank();
1843  tmp_handles[0] = entity;
1844  num_ents = 1;
1845  }
1846 
1847 #ifndef NDEBUG
1848  int tmp_ps = num_ents;
1849 #endif
1850 
1851  // Now add others, with zero handle for now
1852  for( std::set< unsigned int >::iterator sit = procs.begin(); sit != procs.end(); ++sit )
1853  {
1854 #ifndef NDEBUG
1855  if( tmp_ps && std::find( tmp_procs, tmp_procs + tmp_ps, *sit ) != tmp_procs + tmp_ps )
1856  {
1857  std::cerr << "Trouble with something already in shared list on proc " << procConfig.proc_rank()
1858  << ". Entity:" << std::endl;
1859  list_entities( &entity, 1 );
1860  std::cerr << "pstatus = " << (int)pstatus << ", sharedp = " << sharedp << std::endl;
1861  std::cerr << "tmp_ps = ";
1862  for( int i = 0; i < tmp_ps; i++ )
1863  std::cerr << tmp_procs[i] << " ";
1864  std::cerr << std::endl;
1865  std::cerr << "procs = ";
1866  for( std::set< unsigned int >::iterator sit2 = procs.begin(); sit2 != procs.end(); ++sit2 )
1867  std::cerr << *sit2 << " ";
1868  assert( false );
1869  }
1870 #endif
1871  tmp_procs[num_ents] = *sit;
1872  tmp_handles[num_ents] = 0;
1873  num_ents++;
1874  }
1875 
1876  // Put -1 after procs and 0 after handles
1877  if( MAX_SHARING_PROCS > num_ents )
1878  {
1879  tmp_procs[num_ents] = -1;
1880  tmp_handles[num_ents] = 0;
1881  }
1882 
1883  return MB_SUCCESS;
1884 }

References ErrorCode, get_sharing_data(), list_entities(), MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SUCCESS, moab::ProcConfig::proc_rank(), procConfig, PSTATUS_MULTISHARED, PSTATUS_NOT_OWNED, and PSTATUS_SHARED.

Referenced by pack_entities().

◆ check_all_shared_handles() [1/2]

ErrorCode moab::ParallelComm::check_all_shared_handles ( bool  print_em = false)

Call exchange_all_shared_handles, then compare the results with tag data on local shared entities.

Definition at line 8869 of file ParallelComm.cpp.

8870 {
8871  // Get all shared ent data from other procs
8872  std::vector< std::vector< SharedEntityData > > shents( buffProcs.size() ), send_data( buffProcs.size() );
8873 
8874  ErrorCode result;
8875  bool done = false;
8876 
8877  while( !done )
8878  {
8879  result = check_local_shared();
8880  if( MB_SUCCESS != result )
8881  {
8882  done = true;
8883  continue;
8884  }
8885 
8886  result = pack_shared_handles( send_data );
8887  if( MB_SUCCESS != result )
8888  {
8889  done = true;
8890  continue;
8891  }
8892 
8893  result = exchange_all_shared_handles( send_data, shents );
8894  if( MB_SUCCESS != result )
8895  {
8896  done = true;
8897  continue;
8898  }
8899 
8900  if( !shents.empty() ) result = check_my_shared_handles( shents );
8901  done = true;
8902  }
8903 
8904  if( MB_SUCCESS != result && print_em )
8905  {
8906 #ifdef MOAB_HAVE_HDF5
8907  std::ostringstream ent_str;
8908  ent_str << "mesh." << procConfig.proc_rank() << ".h5m";
8909  mbImpl->write_mesh( ent_str.str().c_str() );
8910 #endif
8911  }
8912 
8913  return result;
8914 }

References buffProcs, check_local_shared(), check_my_shared_handles(), ErrorCode, exchange_all_shared_handles(), MB_SUCCESS, mbImpl, pack_shared_handles(), moab::ProcConfig::proc_rank(), procConfig, and moab::Interface::write_mesh().

Referenced by exchange_ghost_cells(), main(), resolve_shared_ents(), and moab::ScdInterface::tag_shared_vertices().

◆ check_all_shared_handles() [2/2]

ErrorCode moab::ParallelComm::check_all_shared_handles ( ParallelComm **  pcs,
int  num_pcs 
)
static

Definition at line 9043 of file ParallelComm.cpp.

9044 {
9045  std::vector< std::vector< std::vector< SharedEntityData > > > shents, send_data;
9046  ErrorCode result = MB_SUCCESS, tmp_result;
9047 
9048  // Get all shared ent data from each proc to all other procs
9049  send_data.resize( num_pcs );
9050  for( int p = 0; p < num_pcs; p++ )
9051  {
9052  tmp_result = pcs[p]->pack_shared_handles( send_data[p] );
9053  if( MB_SUCCESS != tmp_result ) result = tmp_result;
9054  }
9055  if( MB_SUCCESS != result ) return result;
9056 
9057  // Move the data sorted by sending proc to data sorted by receiving proc
9058  shents.resize( num_pcs );
9059  for( int p = 0; p < num_pcs; p++ )
9060  shents[p].resize( pcs[p]->buffProcs.size() );
9061 
9062  for( int p = 0; p < num_pcs; p++ )
9063  {
9064  for( unsigned int idx_p = 0; idx_p < pcs[p]->buffProcs.size(); idx_p++ )
9065  {
9066  // Move send_data[p][to_p] to shents[to_p][idx_p]
9067  int to_p = pcs[p]->buffProcs[idx_p];
9068  int top_idx_p = pcs[to_p]->get_buffers( p );
9069  assert( -1 != top_idx_p );
9070  shents[to_p][top_idx_p] = send_data[p][idx_p];
9071  }
9072  }
9073 
9074  for( int p = 0; p < num_pcs; p++ )
9075  {
9076  std::ostringstream ostr;
9077  ostr << "Processor " << p << " bad entities:";
9078  tmp_result = pcs[p]->check_my_shared_handles( shents[p], ostr.str().c_str() );
9079  if( MB_SUCCESS != tmp_result ) result = tmp_result;
9080  }
9081 
9082  return result;
9083 }

References buffProcs, check_my_shared_handles(), ErrorCode, get_buffers(), MB_SUCCESS, and pack_shared_handles().

◆ check_clean_iface()

ErrorCode moab::ParallelComm::check_clean_iface ( Range &  allsent)
private

Definition at line 6473 of file ParallelComm.cpp.

6474 {
6475  // allsent is all entities I think are on interface; go over them, looking
6476  // for zero-valued handles, and fix any I find
6477 
6478  // Keep lists of entities for which teh sharing data changed, grouped
6479  // by set of sharing procs.
6480  typedef std::map< ProcList, Range > procmap_t;
6481  procmap_t old_procs, new_procs;
6482 
6483  ErrorCode result = MB_SUCCESS;
6484  Range::iterator rit;
6485  Range::reverse_iterator rvit;
6486  unsigned char pstatus;
6487  int nump;
6488  ProcList sharedp;
6489  EntityHandle sharedh[MAX_SHARING_PROCS];
6490  for( rvit = allsent.rbegin(); rvit != allsent.rend(); ++rvit )
6491  {
6492  result = get_sharing_data( *rvit, sharedp.procs, sharedh, pstatus, nump );
6493  MB_CHK_SET_ERR( result, "Failed to get sharing data" );
6494  assert( "Should be shared with at least one other proc" &&
6495  ( nump > 1 || sharedp.procs[0] != (int)procConfig.proc_rank() ) );
6496  assert( nump == MAX_SHARING_PROCS || sharedp.procs[nump] == -1 );
6497 
6498  // Look for first null handle in list
6499  int idx = std::find( sharedh, sharedh + nump, (EntityHandle)0 ) - sharedh;
6500  if( idx == nump ) continue; // All handles are valid
6501 
6502  ProcList old_list( sharedp );
6503  std::sort( old_list.procs, old_list.procs + nump );
6504  old_procs[old_list].insert( *rvit );
6505 
6506  // Remove null handles and corresponding proc ranks from lists
6507  int new_nump = idx;
6508  bool removed_owner = !idx;
6509  for( ++idx; idx < nump; ++idx )
6510  {
6511  if( sharedh[idx] )
6512  {
6513  sharedh[new_nump] = sharedh[idx];
6514  sharedp.procs[new_nump] = sharedp.procs[idx];
6515  ++new_nump;
6516  }
6517  }
6518  sharedp.procs[new_nump] = -1;
6519 
6520  if( removed_owner && new_nump > 1 )
6521  {
6522  // The proc that we choose as the entity owner isn't sharing the
6523  // entity (doesn't have a copy of it). We need to pick a different
6524  // owner. Choose the proc with lowest rank.
6525  idx = std::min_element( sharedp.procs, sharedp.procs + new_nump ) - sharedp.procs;
6526  std::swap( sharedp.procs[0], sharedp.procs[idx] );
6527  std::swap( sharedh[0], sharedh[idx] );
6528  if( sharedp.procs[0] == (int)proc_config().proc_rank() ) pstatus &= ~PSTATUS_NOT_OWNED;
6529  }
6530 
6531  result = set_sharing_data( *rvit, pstatus, nump, new_nump, sharedp.procs, sharedh );
6532  MB_CHK_SET_ERR( result, "Failed to set sharing data in check_clean_iface" );
6533 
6534  if( new_nump > 1 )
6535  {
6536  if( new_nump == 2 )
6537  {
6538  if( sharedp.procs[1] != (int)proc_config().proc_rank() )
6539  {
6540  assert( sharedp.procs[0] == (int)proc_config().proc_rank() );
6541  sharedp.procs[0] = sharedp.procs[1];
6542  }
6543  sharedp.procs[1] = -1;
6544  }
6545  else
6546  {
6547  std::sort( sharedp.procs, sharedp.procs + new_nump );
6548  }
6549  new_procs[sharedp].insert( *rvit );
6550  }
6551  }
6552 
6553  if( old_procs.empty() )
6554  {
6555  assert( new_procs.empty() );
6556  return MB_SUCCESS;
6557  }
6558 
6559  // Update interface sets
6560  procmap_t::iterator pmit;
6561  // std::vector<unsigned char> pstatus_list;
6562  rit = interface_sets().begin();
6563  while( rit != interface_sets().end() )
6564  {
6565  result = get_sharing_data( *rit, sharedp.procs, sharedh, pstatus, nump );
6566  MB_CHK_SET_ERR( result, "Failed to get sharing data for interface set" );
6567  assert( nump != 2 );
6568  std::sort( sharedp.procs, sharedp.procs + nump );
6569  assert( nump == MAX_SHARING_PROCS || sharedp.procs[nump] == -1 );
6570 
6571  pmit = old_procs.find( sharedp );
6572  if( pmit != old_procs.end() )
6573  {
6574  result = mbImpl->remove_entities( *rit, pmit->second );
6575  MB_CHK_SET_ERR( result, "Failed to remove entities from interface set" );
6576  }
6577 
6578  pmit = new_procs.find( sharedp );
6579  if( pmit == new_procs.end() )
6580  {
6581  int count;
6582  result = mbImpl->get_number_entities_by_handle( *rit, count );
6583  MB_CHK_SET_ERR( result, "Failed to get number of entities in interface set" );
6584  if( !count )
6585  {
6586  result = mbImpl->delete_entities( &*rit, 1 );
6587  MB_CHK_SET_ERR( result, "Failed to delete entities from interface set" );
6588  rit = interface_sets().erase( rit );
6589  }
6590  else
6591  {
6592  ++rit;
6593  }
6594  }
6595  else
6596  {
6597  result = mbImpl->add_entities( *rit, pmit->second );
6598  MB_CHK_SET_ERR( result, "Failed to add entities to interface set" );
6599 
6600  // Remove those that we've processed so that we know which ones
6601  // are new.
6602  new_procs.erase( pmit );
6603  ++rit;
6604  }
6605  }
6606 
6607  // Create interface sets for new proc id combinations
6608  std::fill( sharedh, sharedh + MAX_SHARING_PROCS, 0 );
6609  for( pmit = new_procs.begin(); pmit != new_procs.end(); ++pmit )
6610  {
6611  EntityHandle new_set;
6612  result = mbImpl->create_meshset( MESHSET_SET, new_set );
6613  MB_CHK_SET_ERR( result, "Failed to create interface set" );
6614  interfaceSets.insert( new_set );
6615 
6616  // Add entities
6617  result = mbImpl->add_entities( new_set, pmit->second );
6618  MB_CHK_SET_ERR( result, "Failed to add entities to interface set" );
6619  // Tag set with the proc rank(s)
6620  assert( pmit->first.procs[0] >= 0 );
6621  pstatus = PSTATUS_SHARED | PSTATUS_INTERFACE;
6622  if( pmit->first.procs[1] == -1 )
6623  {
6624  int other = pmit->first.procs[0];
6625  assert( other != (int)procConfig.proc_rank() );
6626  result = mbImpl->tag_set_data( sharedp_tag(), &new_set, 1, pmit->first.procs );
6627  MB_CHK_SET_ERR( result, "Failed to tag interface set with procs" );
6628  sharedh[0] = 0;
6629  result = mbImpl->tag_set_data( sharedh_tag(), &new_set, 1, sharedh );
6630  MB_CHK_SET_ERR( result, "Failed to tag interface set with procs" );
6631  if( other < (int)proc_config().proc_rank() ) pstatus |= PSTATUS_NOT_OWNED;
6632  }
6633  else
6634  {
6635  result = mbImpl->tag_set_data( sharedps_tag(), &new_set, 1, pmit->first.procs );
6636  MB_CHK_SET_ERR( result, "Failed to tag interface set with procs" );
6637  result = mbImpl->tag_set_data( sharedhs_tag(), &new_set, 1, sharedh );
6638  MB_CHK_SET_ERR( result, "Failed to tag interface set with procs" );
6639  pstatus |= PSTATUS_MULTISHARED;
6640  if( pmit->first.procs[0] < (int)proc_config().proc_rank() ) pstatus |= PSTATUS_NOT_OWNED;
6641  }
6642 
6643  result = mbImpl->tag_set_data( pstatus_tag(), &new_set, 1, &pstatus );
6644  MB_CHK_SET_ERR( result, "Failed to tag interface set with pstatus" );
6645 
6646  // Set pstatus on all interface entities in set
6647  result = mbImpl->tag_clear_data( pstatus_tag(), pmit->second, &pstatus );
6648  MB_CHK_SET_ERR( result, "Failed to tag interface entities with pstatus" );
6649  }
6650 
6651  return MB_SUCCESS;
6652 }

References moab::Interface::add_entities(), moab::Range::begin(), moab::Interface::create_meshset(), moab::Interface::delete_entities(), moab::Range::erase(), ErrorCode, moab::Interface::get_number_entities_by_handle(), get_sharing_data(), moab::Range::insert(), interface_sets(), interfaceSets, MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, MESHSET_SET, proc_config(), moab::ProcConfig::proc_rank(), procConfig, moab::ProcList::procs, PSTATUS_INTERFACE, PSTATUS_MULTISHARED, PSTATUS_NOT_OWNED, PSTATUS_SHARED, pstatus_tag(), moab::Range::rbegin(), moab::Interface::remove_entities(), moab::Range::rend(), set_sharing_data(), sharedh_tag(), sharedhs_tag(), sharedp_tag(), sharedps_tag(), moab::Interface::tag_clear_data(), and moab::Interface::tag_set_data().

Referenced by exchange_ghost_cells().

◆ check_global_ids()

ErrorCode moab::ParallelComm::check_global_ids ( EntityHandle  this_set,
const int  dimension,
const int  start_id = 1,
const bool  largest_dim_only = true,
const bool  parallel = true,
const bool  owned_only = false 
)

check for global ids; based only on tag handle being there or not; if it's not there, create them for the specified dimensions

Parameters
owned_onlyIf true, do not get global IDs for non-owned entities from remote processors.

Definition at line 5731 of file ParallelComm.cpp.

5737 {
5738  // Global id tag
5739  Tag gid_tag = mbImpl->globalId_tag();
5740  int def_val = -1;
5741  Range dum_range;
5742 
5743  void* tag_ptr = &def_val;
5744  ErrorCode result = mbImpl->get_entities_by_type_and_tag( this_set, MBVERTEX, &gid_tag, &tag_ptr, 1, dum_range );
5745  MB_CHK_SET_ERR( result, "Failed to get entities by MBVERTEX type and gid tag" );
5746 
5747  if( !dum_range.empty() )
5748  {
5749  // Just created it, so we need global ids
5750  result = assign_global_ids( this_set, dimension, start_id, largest_dim_only, parallel, owned_only );
5751  MB_CHK_SET_ERR( result, "Failed assigning global ids" );
5752  }
5753 
5754  return MB_SUCCESS;
5755 }

References assign_global_ids(), moab::Range::empty(), ErrorCode, moab::Interface::get_entities_by_type_and_tag(), moab::Interface::globalId_tag(), MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, and MBVERTEX.

Referenced by moab::ReadParallel::load_file().

◆ check_local_shared()

ErrorCode moab::ParallelComm::check_local_shared ( )

Definition at line 8916 of file ParallelComm.cpp.

8917 {
8918  // Do some checks on shared entities to make sure things look
8919  // consistent
8920 
8921  // Check that non-vertex shared entities are shared by same procs as all
8922  // their vertices
8923  // std::pair<Range::const_iterator,Range::const_iterator> vert_it =
8924  // sharedEnts.equal_range(MBVERTEX);
8925  std::vector< EntityHandle > dum_connect;
8926  const EntityHandle* connect;
8927  int num_connect;
8928  int tmp_procs[MAX_SHARING_PROCS];
8929  EntityHandle tmp_hs[MAX_SHARING_PROCS];
8930  std::set< int > tmp_set, vset;
8931  int num_ps;
8932  ErrorCode result;
8933  unsigned char pstat;
8934  std::vector< EntityHandle > bad_ents;
8935  std::vector< std::string > errors;
8936 
8937  std::set< EntityHandle >::iterator vit;
8938  for( vit = sharedEnts.begin(); vit != sharedEnts.end(); ++vit )
8939  {
8940  // Get sharing procs for this ent
8941  result = get_sharing_data( *vit, tmp_procs, tmp_hs, pstat, num_ps );
8942  if( MB_SUCCESS != result )
8943  {
8944  bad_ents.push_back( *vit );
8945  errors.push_back( std::string( "Failure getting sharing data." ) );
8946  continue;
8947  }
8948 
8949  bool bad = false;
8950  // Entity must be shared
8951  if( !( pstat & PSTATUS_SHARED ) )
8952  errors.push_back( std::string( "Entity should be shared but isn't." ) ), bad = true;
8953 
8954  // If entity is not owned this must not be first proc
8955  if( pstat & PSTATUS_NOT_OWNED && tmp_procs[0] == (int)procConfig.proc_rank() )
8956  errors.push_back( std::string( "Entity not owned but is first proc." ) ), bad = true;
8957 
8958  // If entity is owned and multishared, this must be first proc
8959  if( !( pstat & PSTATUS_NOT_OWNED ) && pstat & PSTATUS_MULTISHARED &&
8960  ( tmp_procs[0] != (int)procConfig.proc_rank() || tmp_hs[0] != *vit ) )
8961  errors.push_back( std::string( "Entity owned and multishared but not first proc or not first handle." ) ),
8962  bad = true;
8963 
8964  if( bad )
8965  {
8966  bad_ents.push_back( *vit );
8967  continue;
8968  }
8969 
8970  EntityType type = mbImpl->type_from_handle( *vit );
8971  if( type == MBVERTEX || type == MBENTITYSET ) continue;
8972 
8973  // Copy element's procs to vset and save size
8974  int orig_ps = num_ps;
8975  vset.clear();
8976  std::copy( tmp_procs, tmp_procs + num_ps, std::inserter( vset, vset.begin() ) );
8977 
8978  // Get vertices for this ent and intersection of sharing procs
8979  result = mbImpl->get_connectivity( *vit, connect, num_connect, false, &dum_connect );
8980  if( MB_SUCCESS != result )
8981  {
8982  bad_ents.push_back( *vit );
8983  errors.push_back( std::string( "Failed to get connectivity." ) );
8984  continue;
8985  }
8986 
8987  for( int i = 0; i < num_connect; i++ )
8988  {
8989  result = get_sharing_data( connect[i], tmp_procs, NULL, pstat, num_ps );
8990  if( MB_SUCCESS != result )
8991  {
8992  bad_ents.push_back( *vit );
8993  continue;
8994  }
8995  if( !num_ps )
8996  {
8997  vset.clear();
8998  break;
8999  }
9000  std::sort( tmp_procs, tmp_procs + num_ps );
9001  tmp_set.clear();
9002  std::set_intersection( tmp_procs, tmp_procs + num_ps, vset.begin(), vset.end(),
9003  std::inserter( tmp_set, tmp_set.end() ) );
9004  vset.swap( tmp_set );
9005  if( vset.empty() ) break;
9006  }
9007 
9008  // Intersect them; should be the same size as orig_ps
9009  tmp_set.clear();
9010  std::set_intersection( tmp_procs, tmp_procs + num_ps, vset.begin(), vset.end(),
9011  std::inserter( tmp_set, tmp_set.end() ) );
9012  if( orig_ps != (int)tmp_set.size() )
9013  {
9014  errors.push_back( std::string( "Vertex proc set not same size as entity proc set." ) );
9015  bad_ents.push_back( *vit );
9016  for( int i = 0; i < num_connect; i++ )
9017  {
9018  bad_ents.push_back( connect[i] );
9019  errors.push_back( std::string( "vertex in connect" ) );
9020  }
9021  }
9022  }
9023 
9024  if( !bad_ents.empty() )
9025  {
9026  std::cout << "Found bad entities in check_local_shared, proc rank " << procConfig.proc_rank() << ","
9027  << std::endl;
9028  std::vector< std::string >::iterator sit;
9029  std::vector< EntityHandle >::iterator rit;
9030  for( rit = bad_ents.begin(), sit = errors.begin(); rit != bad_ents.end(); ++rit, ++sit )
9031  {
9032  list_entities( &( *rit ), 1 );
9033  std::cout << "Reason: " << *sit << std::endl;
9034  }
9035  return MB_FAILURE;
9036  }
9037 
9038  // To do: check interface sets
9039 
9040  return MB_SUCCESS;
9041 }

References ErrorCode, moab::Interface::get_connectivity(), get_sharing_data(), list_entities(), MAX_SHARING_PROCS, MB_SUCCESS, MBENTITYSET, mbImpl, MBVERTEX, moab::ProcConfig::proc_rank(), procConfig, PSTATUS_MULTISHARED, PSTATUS_NOT_OWNED, PSTATUS_SHARED, sharedEnts, and moab::Interface::type_from_handle().

Referenced by check_all_shared_handles().

◆ check_my_shared_handles()

ErrorCode moab::ParallelComm::check_my_shared_handles ( std::vector< std::vector< SharedEntityData > > &  shents,
const char *  prefix = NULL 
)

Definition at line 9085 of file ParallelComm.cpp.

9087 {
9088  // Now check against what I think data should be
9089  // Get all shared entities
9090  ErrorCode result;
9091  Range all_shared;
9092  std::copy( sharedEnts.begin(), sharedEnts.end(), range_inserter( all_shared ) );
9093  std::vector< EntityHandle > dum_vec;
9094  all_shared.erase( all_shared.upper_bound( MBPOLYHEDRON ), all_shared.end() );
9095 
9096  Range bad_ents, local_shared;
9097  std::vector< SharedEntityData >::iterator vit;
9098  unsigned char tmp_pstat;
9099  for( unsigned int i = 0; i < shents.size(); i++ )
9100  {
9101  int other_proc = buffProcs[i];
9102  result = get_shared_entities( other_proc, local_shared );
9103  if( MB_SUCCESS != result ) return result;
9104  for( vit = shents[i].begin(); vit != shents[i].end(); ++vit )
9105  {
9106  EntityHandle localh = vit->local, remoteh = vit->remote, dumh;
9107  local_shared.erase( localh );
9108  result = get_remote_handles( true, &localh, &dumh, 1, other_proc, dum_vec );
9109  if( MB_SUCCESS != result || dumh != remoteh ) bad_ents.insert( localh );
9110  result = get_pstatus( localh, tmp_pstat );
9111  if( MB_SUCCESS != result || ( !( tmp_pstat & PSTATUS_NOT_OWNED ) && (unsigned)vit->owner != rank() ) ||
9112  ( tmp_pstat & PSTATUS_NOT_OWNED && (unsigned)vit->owner == rank() ) )
9113  bad_ents.insert( localh );
9114  }
9115 
9116  if( !local_shared.empty() ) bad_ents.merge( local_shared );
9117  }
9118 
9119  if( !bad_ents.empty() )
9120  {
9121  if( prefix ) std::cout << prefix << std::endl;
9122  list_entities( bad_ents );
9123  return MB_FAILURE;
9124  }
9125  else
9126  return MB_SUCCESS;
9127 }

References buffProcs, moab::Range::empty(), moab::Range::end(), moab::Range::erase(), ErrorCode, get_pstatus(), get_remote_handles(), get_shared_entities(), moab::Range::insert(), list_entities(), MB_SUCCESS, MBPOLYHEDRON, moab::Range::merge(), PSTATUS_NOT_OWNED, rank(), sharedEnts, and moab::Range::upper_bound().

Referenced by check_all_shared_handles().

◆ check_sent_ents()

ErrorCode moab::ParallelComm::check_sent_ents ( Range &  allsent)
private

check entities to make sure there are no zero-valued remote handles where they shouldn't be

Definition at line 7600 of file ParallelComm.cpp.

7601 {
7602  // Check entities to make sure there are no zero-valued remote handles
7603  // where they shouldn't be
7604  std::vector< unsigned char > pstat( allsent.size() );
7605  ErrorCode result = mbImpl->tag_get_data( pstatus_tag(), allsent, pstat.data() );
7606  MB_CHK_SET_ERR( result, "Failed to get pstatus tag data" );
7607  std::vector< EntityHandle > handles( allsent.size() );
7608  result = mbImpl->tag_get_data( sharedh_tag(), allsent, handles.data() );
7609  MB_CHK_SET_ERR( result, "Failed to get sharedh tag data" );
7610  std::vector< int > procs( allsent.size() );
7611  result = mbImpl->tag_get_data( sharedp_tag(), allsent, procs.data() );
7612  MB_CHK_SET_ERR( result, "Failed to get sharedp tag data" );
7613 
7614  Range bad_entities;
7615 
7616  Range::iterator rit;
7617  unsigned int i;
7618  EntityHandle dum_hs[MAX_SHARING_PROCS];
7619  int dum_ps[MAX_SHARING_PROCS];
7620 
7621  for( rit = allsent.begin(), i = 0; rit != allsent.end(); ++rit, i++ )
7622  {
7623  if( -1 != procs[i] && 0 == handles[i] )
7624  bad_entities.insert( *rit );
7625  else
7626  {
7627  // Might be multi-shared...
7628  result = mbImpl->tag_get_data( sharedps_tag(), &( *rit ), 1, dum_ps );
7629  if( MB_TAG_NOT_FOUND == result )
7630  continue;
7631  else if( MB_SUCCESS != result )
7632  MB_SET_ERR( result, "Failed to get sharedps tag data" );
7633  result = mbImpl->tag_get_data( sharedhs_tag(), &( *rit ), 1, dum_hs );
7634  MB_CHK_SET_ERR( result, "Failed to get sharedhs tag data" );
7635 
7636  // Find first non-set proc
7637  int* ns_proc = std::find( dum_ps, dum_ps + MAX_SHARING_PROCS, -1 );
7638  int num_procs = ns_proc - dum_ps;
7639  assert( num_procs <= MAX_SHARING_PROCS );
7640  // Now look for zero handles in active part of dum_hs
7641  EntityHandle* ns_handle = std::find( dum_hs, dum_hs + num_procs, 0 );
7642  int num_handles = ns_handle - dum_hs;
7643  assert( num_handles <= num_procs );
7644  if( num_handles != num_procs ) bad_entities.insert( *rit );
7645  }
7646  }
7647 
7648  return MB_SUCCESS;
7649 }

References moab::Range::begin(), moab::Range::end(), ErrorCode, moab::Range::insert(), MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SET_ERR, MB_SUCCESS, MB_TAG_NOT_FOUND, mbImpl, pstatus_tag(), sharedh_tag(), sharedhs_tag(), sharedp_tag(), sharedps_tag(), moab::Range::size(), and moab::Interface::tag_get_data().

Referenced by exchange_ghost_cells(), and exchange_owned_mesh().

◆ clean_shared_tags()

ErrorCode moab::ParallelComm::clean_shared_tags ( std::vector< Range * > &  exchange_ents)

Definition at line 9173 of file ParallelComm.cpp.

9174 {
9175  for( unsigned int i = 0; i < exchange_ents.size(); i++ )
9176  {
9177  Range* ents = exchange_ents[i];
9178  int num_ents = ents->size();
9179  Range::iterator it = ents->begin();
9180 
9181  for( int n = 0; n < num_ents; n++ )
9182  {
9183  int sharing_proc;
9184  ErrorCode result = mbImpl->tag_get_data( sharedp_tag(), &( *ents->begin() ), 1, &sharing_proc );
9185  if( result != MB_TAG_NOT_FOUND && sharing_proc == -1 )
9186  {
9187  result = mbImpl->tag_delete_data( sharedp_tag(), &( *it ), 1 );
9188  MB_CHK_SET_ERR( result, "Failed to delete sharedp tag data" );
9189  result = mbImpl->tag_delete_data( sharedh_tag(), &( *it ), 1 );
9190  MB_CHK_SET_ERR( result, "Failed to delete sharedh tag data" );
9191  result = mbImpl->tag_delete_data( pstatus_tag(), &( *it ), 1 );
9192  MB_CHK_SET_ERR( result, "Failed to delete pstatus tag data" );
9193  }
9194  ++it;
9195  }
9196  }
9197 
9198  return MB_SUCCESS;
9199 }

References moab::Range::begin(), ErrorCode, MB_CHK_SET_ERR, MB_SUCCESS, MB_TAG_NOT_FOUND, mbImpl, pstatus_tag(), sharedh_tag(), sharedp_tag(), moab::Range::size(), moab::Interface::tag_delete_data(), and moab::Interface::tag_get_data().

◆ collective_sync_partition()

ErrorCode moab::ParallelComm::collective_sync_partition ( )

Definition at line 8592 of file ParallelComm.cpp.

8593 {
8594  int count = partition_sets().size();
8595  globalPartCount = 0;
8596  int err = MPI_Allreduce( &count, &globalPartCount, 1, MPI_INT, MPI_SUM, proc_config().proc_comm() );
8597  return err ? MB_FAILURE : MB_SUCCESS;
8598 }

References globalPartCount, MB_SUCCESS, partition_sets(), proc_config(), and moab::Range::size().

◆ comm()

MPI_Comm moab::ParallelComm::comm ( ) const
inline
Examples
ExchangeHalos.cpp.

Definition at line 652 of file ParallelComm.hpp.

653  {
654  return proc_config().proc_comm();
655  }

References moab::ProcConfig::proc_comm(), and proc_config().

Referenced by gather_data(), get_pcomm(), moab::ParallelMergeMesh::GetGlobalBox(), iMOAB_ReceiveMesh(), iMOAB_SendMesh(), iMOAB_UpdateMeshInfo(), main(), perform_laplacian_smoothing(), perform_lloyd_relaxation(), moab::LloydSmoother::perform_smooth(), moab::ParallelMergeMesh::PerformMerge(), moab::ParCommGraph::receive_comm_graph(), moab::ParCommGraph::send_graph_partition(), ToolContext::timer_pop(), and RuntimeContext::timer_pop().

◆ correct_thin_ghost_layers()

ErrorCode moab::ParallelComm::correct_thin_ghost_layers ( )

Definition at line 9687 of file ParallelComm.cpp.

9688 {
9689 
9690  // Get all shared ent data from other procs
9691  std::vector< std::vector< SharedEntityData > > shents( buffProcs.size() ), send_data( buffProcs.size() );
9692 
9693  // will work only on multi-shared tags sharedps_tag(), sharedhs_tag();
9694 
9695  /*
9696  * domain0 | domain1 | domain2 | domain3
9697  * vertices from domain 1 and 2 are visible from both 0 and 3, but
9698  * domain 0 might not have info about multi-sharing from domain 3
9699  * so we will force that domain 0 vertices owned by 1 and 2 have information
9700  * about the domain 3 sharing
9701  *
9702  * SharedEntityData will have :
9703  * struct SharedEntityData {
9704  EntityHandle local; // this is same meaning, for the proc we sent to, it is local
9705  EntityHandle remote; // this will be the far away handle that will need to be added
9706  EntityID owner; // this will be the remote proc
9707  };
9708  // so we need to add data like this:
9709  a multishared entity owned by proc x will have data like
9710  multishared procs: proc x, a, b, c
9711  multishared handles: h1, h2, h3, h4
9712  we will need to send data from proc x like this:
9713  to proc a we will send
9714  (h2, h3, b), (h2, h4, c)
9715  to proc b we will send
9716  (h3, h2, a), (h3, h4, c)
9717  to proc c we will send
9718  (h4, h2, a), (h4, h3, b)
9719  *
9720  */
9721 
9722  ErrorCode result = MB_SUCCESS;
9723  int ent_procs[MAX_SHARING_PROCS + 1];
9724  EntityHandle handles[MAX_SHARING_PROCS + 1];
9725  int num_sharing;
9726  SharedEntityData tmp;
9727 
9728  for( std::set< EntityHandle >::iterator i = sharedEnts.begin(); i != sharedEnts.end(); ++i )
9729  {
9730 
9731  unsigned char pstat;
9732  result = get_sharing_data( *i, ent_procs, handles, pstat, num_sharing );
9733  MB_CHK_SET_ERR( result, "can't get sharing data" );
9734  if( !( pstat & PSTATUS_MULTISHARED ) ||
9735  num_sharing <= 2 ) // if not multishared, skip, it should have no problems
9736  continue;
9737  // we should skip the ones that are not owned locally
9738  // the owned ones will have the most multi-shared info, because the info comes from other
9739  // remote processors
9740  if( pstat & PSTATUS_NOT_OWNED ) continue;
9741  for( int j = 1; j < num_sharing; j++ )
9742  {
9743  // we will send to proc
9744  int send_to_proc = ent_procs[j]; //
9745  tmp.local = handles[j];
9746  int ind = get_buffers( send_to_proc );
9747  assert( -1 != ind ); // THIS SHOULD NEVER HAPPEN
9748  for( int k = 1; k < num_sharing; k++ )
9749  {
9750  // do not send to self proc
9751  if( j == k ) continue;
9752  tmp.remote = handles[k]; // this will be the handle of entity on proc
9753  tmp.owner = ent_procs[k];
9754  send_data[ind].push_back( tmp );
9755  }
9756  }
9757  }
9758 
9759  result = exchange_all_shared_handles( send_data, shents );
9760  MB_CHK_ERR( result );
9761 
9762  // loop over all shents and add if vertex type, add if missing
9763  for( size_t i = 0; i < shents.size(); i++ )
9764  {
9765  std::vector< SharedEntityData >& shEnts = shents[i];
9766  for( size_t j = 0; j < shEnts.size(); j++ )
9767  {
9768  tmp = shEnts[j];
9769  // basically, check the shared data for tmp.local entity
9770  // it should have inside the tmp.owner and tmp.remote
9771  EntityHandle eh = tmp.local;
9772  unsigned char pstat;
9773  result = get_sharing_data( eh, ent_procs, handles, pstat, num_sharing );
9774  MB_CHK_SET_ERR( result, "can't get sharing data" );
9775  // see if the proc tmp.owner is in the list of ent_procs; if not, we have to increase
9776  // handles, and ent_procs; and set
9777 
9778  int proc_remote = tmp.owner; //
9779  if( std::find( ent_procs, ent_procs + num_sharing, proc_remote ) == ent_procs + num_sharing )
9780  {
9781  // so we did not find on proc
9782 #ifndef NDEBUG
9783  if( myDebug->get_verbosity() == 3 )
9784  std::cout << "THIN GHOST: we did not find on proc " << rank() << " for shared ent " << eh
9785  << " the proc " << proc_remote << "\n";
9786 #endif
9787  // increase num_sharing, and set the multi-shared tags
9788  if( num_sharing >= MAX_SHARING_PROCS ) return MB_FAILURE;
9789  handles[num_sharing] = tmp.remote;
9790  handles[num_sharing + 1] = 0; // end of list
9791  ent_procs[num_sharing] = tmp.owner;
9792  ent_procs[num_sharing + 1] = -1; // this should be already set
9793  result = mbImpl->tag_set_data( sharedps_tag(), &eh, 1, ent_procs );
9794  MB_CHK_SET_ERR( result, "Failed to set sharedps tag data" );
9795  result = mbImpl->tag_set_data( sharedhs_tag(), &eh, 1, handles );
9796  MB_CHK_SET_ERR( result, "Failed to set sharedhs tag data" );
9797  if( 2 == num_sharing ) // it means the sharedp and sharedh tags were set with a
9798  // value non default
9799  {
9800  // so entity eh was simple shared before, we need to set those dense tags back
9801  // to default
9802  // values
9803  EntityHandle zero = 0;
9804  int no_proc = -1;
9805  result = mbImpl->tag_set_data( sharedp_tag(), &eh, 1, &no_proc );
9806  MB_CHK_SET_ERR( result, "Failed to set sharedp tag data" );
9807  result = mbImpl->tag_set_data( sharedh_tag(), &eh, 1, &zero );
9808  MB_CHK_SET_ERR( result, "Failed to set sharedh tag data" );
9809  // also, add multishared pstatus tag
9810  // also add multishared status to pstatus
9811  pstat = pstat | PSTATUS_MULTISHARED;
9812  result = mbImpl->tag_set_data( pstatus_tag(), &eh, 1, &pstat );
9813  MB_CHK_SET_ERR( result, "Failed to set pstatus tag data" );
9814  }
9815  }
9816  }
9817  }
9818  return MB_SUCCESS;
9819 }

References buffProcs, ErrorCode, exchange_all_shared_handles(), get_buffers(), get_sharing_data(), moab::DebugOutput::get_verbosity(), moab::ParallelComm::SharedEntityData::local, MAX_SHARING_PROCS, MB_CHK_ERR, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, myDebug, moab::ParallelComm::SharedEntityData::owner, PSTATUS_MULTISHARED, PSTATUS_NOT_OWNED, pstatus_tag(), rank(), moab::ParallelComm::SharedEntityData::remote, sharedEnts, sharedh_tag(), sharedhs_tag(), sharedp_tag(), sharedps_tag(), and moab::Interface::tag_set_data().

Referenced by iMOAB_DetermineGhostEntities(), and moab::ReadParallel::load_file().

◆ create_iface_pc_links()

ErrorCode moab::ParallelComm::create_iface_pc_links ( )
private

Definition at line 5262 of file ParallelComm.cpp.

5263 {
5264  // Now that we've resolved the entities in the iface sets,
5265  // set parent/child links between the iface sets
5266 
5267  // First tag all entities in the iface sets
5268  Tag tmp_iface_tag;
5269  EntityHandle tmp_iface_set = 0;
5270  ErrorCode result = mbImpl->tag_get_handle( "__tmp_iface", 1, MB_TYPE_HANDLE, tmp_iface_tag,
5271  MB_TAG_DENSE | MB_TAG_CREAT, &tmp_iface_set );
5272  MB_CHK_SET_ERR( result, "Failed to create temporary interface set tag" );
5273 
5274  Range iface_ents;
5275  std::vector< EntityHandle > tag_vals;
5276  Range::iterator rit;
5277 
5278  for( rit = interfaceSets.begin(); rit != interfaceSets.end(); ++rit )
5279  {
5280  // tag entities with interface set
5281  iface_ents.clear();
5282  result = mbImpl->get_entities_by_handle( *rit, iface_ents );
5283  MB_CHK_SET_ERR( result, "Failed to get entities in interface set" );
5284 
5285  if( iface_ents.empty() ) continue;
5286 
5287  tag_vals.resize( iface_ents.size() );
5288  std::fill( tag_vals.begin(), tag_vals.end(), *rit );
5289  result = mbImpl->tag_set_data( tmp_iface_tag, iface_ents, tag_vals.data() );
5290  MB_CHK_SET_ERR( result, "Failed to tag iface entities with interface set" );
5291  }
5292 
5293  // Now go back through interface sets and add parent/child links
5294  Range tmp_ents2;
5295  for( int d = 2; d >= 0; d-- )
5296  {
5297  for( rit = interfaceSets.begin(); rit != interfaceSets.end(); ++rit )
5298  {
5299  // Get entities on this interface
5300  iface_ents.clear();
5301  result = mbImpl->get_entities_by_handle( *rit, iface_ents, true );
5302  MB_CHK_SET_ERR( result, "Failed to get entities by handle" );
5303  if( iface_ents.empty() || mbImpl->dimension_from_handle( *iface_ents.rbegin() ) != d ) continue;
5304 
5305  // Get higher-dimensional entities and their interface sets
5306  result = mbImpl->get_adjacencies( &( *iface_ents.begin() ), 1, d + 1, false, tmp_ents2 );
5307  MB_CHK_SET_ERR( result, "Failed to get adjacencies for interface sets" );
5308  tag_vals.resize( tmp_ents2.size() );
5309  result = mbImpl->tag_get_data( tmp_iface_tag, tmp_ents2, tag_vals.data() );
5310  MB_CHK_SET_ERR( result, "Failed to get tmp iface tag for interface sets" );
5311 
5312  // Go through and for any on interface make it a parent
5313  EntityHandle last_set = 0;
5314  for( unsigned int i = 0; i < tag_vals.size(); i++ )
5315  {
5316  if( tag_vals[i] && tag_vals[i] != last_set )
5317  {
5318  result = mbImpl->add_parent_child( tag_vals[i], *rit );
5319  MB_CHK_SET_ERR( result, "Failed to add parent/child link for interface set" );
5320  last_set = tag_vals[i];
5321  }
5322  }
5323  }
5324  }
5325 
5326  // Delete the temporary tag
5327  result = mbImpl->tag_delete( tmp_iface_tag );
5328  MB_CHK_SET_ERR( result, "Failed to delete tmp iface tag" );
5329 
5330  return MB_SUCCESS;
5331 }

References moab::Interface::add_parent_child(), moab::Range::begin(), moab::Range::clear(), moab::Interface::dimension_from_handle(), moab::Range::empty(), moab::Range::end(), ErrorCode, moab::Interface::get_adjacencies(), moab::Interface::get_entities_by_handle(), interfaceSets, MB_CHK_SET_ERR, MB_SUCCESS, MB_TAG_CREAT, MB_TAG_DENSE, MB_TYPE_HANDLE, mbImpl, moab::Range::rbegin(), moab::Range::size(), moab::Interface::tag_delete(), moab::Interface::tag_get_data(), moab::Interface::tag_get_handle(), and moab::Interface::tag_set_data().

Referenced by resolve_shared_ents(), and moab::ParallelMergeMesh::TagSharedElements().

◆ create_interface_sets() [1/2]

ErrorCode moab::ParallelComm::create_interface_sets ( EntityHandle  this_set,
int  resolve_dim,
int  shared_dim 
)

Definition at line 5137 of file ParallelComm.cpp.

5138 {
5139  std::map< std::vector< int >, std::vector< EntityHandle > > proc_nvecs;
5140 
5141  // Build up the list of shared entities
5142  int procs[MAX_SHARING_PROCS];
5143  EntityHandle handles[MAX_SHARING_PROCS];
5144  ErrorCode result;
5145  int nprocs;
5146  unsigned char pstat;
5147  for( std::set< EntityHandle >::iterator vit = sharedEnts.begin(); vit != sharedEnts.end(); ++vit )
5148  {
5149  if( shared_dim != -1 && mbImpl->dimension_from_handle( *vit ) > shared_dim ) continue;
5150  result = get_sharing_data( *vit, procs, handles, pstat, nprocs );
5151  MB_CHK_SET_ERR( result, "Failed to get sharing data" );
5152  std::sort( procs, procs + nprocs );
5153  std::vector< int > tmp_procs( procs, procs + nprocs );
5154  assert( tmp_procs.size() != 2 );
5155  proc_nvecs[tmp_procs].push_back( *vit );
5156  }
5157 
5158  Skinner skinner( mbImpl );
5159  Range skin_ents[4];
5160  result = mbImpl->get_entities_by_dimension( this_set, resolve_dim, skin_ents[resolve_dim] );
5161  MB_CHK_SET_ERR( result, "Failed to get skin entities by dimension" );
5162  result =
5163  skinner.find_skin( this_set, skin_ents[resolve_dim], false, skin_ents[resolve_dim - 1], 0, true, true, true );
5164  MB_CHK_SET_ERR( result, "Failed to find skin" );
5165  if( shared_dim > 1 )
5166  {
5167  result = mbImpl->get_adjacencies( skin_ents[resolve_dim - 1], resolve_dim - 2, true, skin_ents[resolve_dim - 2],
5168  Interface::UNION );
5169  MB_CHK_SET_ERR( result, "Failed to get skin adjacencies" );
5170  }
5171 
5172  result = get_proc_nvecs( resolve_dim, shared_dim, skin_ents, proc_nvecs );
5173 
5174  return create_interface_sets( proc_nvecs );
5175 }

References create_interface_sets(), moab::Interface::dimension_from_handle(), ErrorCode, moab::Skinner::find_skin(), moab::Interface::get_adjacencies(), moab::Interface::get_entities_by_dimension(), get_proc_nvecs(), get_sharing_data(), MAX_SHARING_PROCS, MB_CHK_SET_ERR, mbImpl, sharedEnts, and moab::Interface::UNION.

◆ create_interface_sets() [2/2]

ErrorCode moab::ParallelComm::create_interface_sets ( std::map< std::vector< int >, std::vector< EntityHandle > > &  proc_nvecs)

Definition at line 5177 of file ParallelComm.cpp.

5178 {
5179  if( proc_nvecs.empty() ) return MB_SUCCESS;
5180 
5181  int proc_ids[MAX_SHARING_PROCS];
5182  EntityHandle proc_handles[MAX_SHARING_PROCS];
5183  Tag shp_tag, shps_tag, shh_tag, shhs_tag, pstat_tag;
5184  ErrorCode result = get_shared_proc_tags( shp_tag, shps_tag, shh_tag, shhs_tag, pstat_tag );
5185  MB_CHK_SET_ERR( result, "Failed to get shared proc tags in create_interface_sets" );
5186  Range::iterator rit;
5187 
5188  // Create interface sets, tag them, and tag their contents with iface set tag
5189  std::vector< unsigned char > pstatus;
5190  for( std::map< std::vector< int >, std::vector< EntityHandle > >::iterator vit = proc_nvecs.begin();
5191  vit != proc_nvecs.end(); ++vit )
5192  {
5193  // Create the set
5194  EntityHandle new_set;
5195  result = mbImpl->create_meshset( MESHSET_SET, new_set );
5196  MB_CHK_SET_ERR( result, "Failed to create interface set" );
5197  interfaceSets.insert( new_set );
5198 
5199  // Add entities
5200  assert( !vit->second.empty() );
5201  result = mbImpl->add_entities( new_set, ( vit->second ).data(), ( vit->second ).size() );
5202  MB_CHK_SET_ERR( result, "Failed to add entities to interface set" );
5203  // Tag set with the proc rank(s)
5204  if( vit->first.size() == 1 )
5205  {
5206  assert( ( vit->first )[0] != (int)procConfig.proc_rank() );
5207  result = mbImpl->tag_set_data( shp_tag, &new_set, 1, ( vit->first ).data() );
5208  MB_CHK_SET_ERR( result, "Failed to tag interface set with procs" );
5209  proc_handles[0] = 0;
5210  result = mbImpl->tag_set_data( shh_tag, &new_set, 1, proc_handles );
5211  MB_CHK_SET_ERR( result, "Failed to tag interface set with procs" );
5212  }
5213  else
5214  {
5215  // Pad tag data out to MAX_SHARING_PROCS with -1
5216  if( vit->first.size() > MAX_SHARING_PROCS )
5217  {
5218  std::cerr << "Exceeded MAX_SHARING_PROCS for " << CN::EntityTypeName( TYPE_FROM_HANDLE( new_set ) )
5219  << ' ' << ID_FROM_HANDLE( new_set ) << " on process " << proc_config().proc_rank()
5220  << std::endl;
5221  std::cerr.flush();
5222  MPI_Abort( proc_config().proc_comm(), 66 );
5223  }
5224  // assert(vit->first.size() <= MAX_SHARING_PROCS);
5225  std::copy( vit->first.begin(), vit->first.end(), proc_ids );
5226  std::fill( proc_ids + vit->first.size(), proc_ids + MAX_SHARING_PROCS, -1 );
5227  result = mbImpl->tag_set_data( shps_tag, &new_set, 1, proc_ids );
5228  MB_CHK_SET_ERR( result, "Failed to tag interface set with procs" );
5229  unsigned int ind = std::find( proc_ids, proc_ids + vit->first.size(), procConfig.proc_rank() ) - proc_ids;
5230  assert( ind < vit->first.size() );
5231  std::fill( proc_handles, proc_handles + MAX_SHARING_PROCS, 0 );
5232  proc_handles[ind] = new_set;
5233  result = mbImpl->tag_set_data( shhs_tag, &new_set, 1, proc_handles );
5234  MB_CHK_SET_ERR( result, "Failed to tag interface set with procs" );
5235  }
5236 
5237  // Get the owning proc, then set the pstatus tag on iface set
5238  int min_proc = ( vit->first )[0];
5239  unsigned char pval = ( PSTATUS_SHARED | PSTATUS_INTERFACE );
5240  if( min_proc < (int)procConfig.proc_rank() ) pval |= PSTATUS_NOT_OWNED;
5241  if( vit->first.size() > 1 ) pval |= PSTATUS_MULTISHARED;
5242  result = mbImpl->tag_set_data( pstat_tag, &new_set, 1, &pval );
5243  MB_CHK_SET_ERR( result, "Failed to tag interface set with pstatus" );
5244 
5245  // Tag the vertices with the same thing
5246  pstatus.clear();
5247  std::vector< EntityHandle > verts;
5248  for( std::vector< EntityHandle >::iterator v2it = ( vit->second ).begin(); v2it != ( vit->second ).end();
5249  ++v2it )
5250  if( mbImpl->type_from_handle( *v2it ) == MBVERTEX ) verts.push_back( *v2it );
5251  pstatus.resize( verts.size(), pval );
5252  if( !verts.empty() )
5253  {
5254  result = mbImpl->tag_set_data( pstat_tag, verts.data(), verts.size(), pstatus.data() );
5255  MB_CHK_SET_ERR( result, "Failed to tag interface set vertices with pstatus" );
5256  }
5257  }
5258 
5259  return MB_SUCCESS;
5260 }

References moab::Interface::add_entities(), moab::Interface::create_meshset(), moab::CN::EntityTypeName(), ErrorCode, moab::GeomUtil::first(), get_shared_proc_tags(), moab::ID_FROM_HANDLE(), moab::Range::insert(), interfaceSets, MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, MBVERTEX, MESHSET_SET, proc_config(), moab::ProcConfig::proc_rank(), procConfig, PSTATUS_INTERFACE, PSTATUS_MULTISHARED, PSTATUS_NOT_OWNED, PSTATUS_SHARED, moab::Interface::tag_set_data(), moab::Interface::type_from_handle(), and moab::TYPE_FROM_HANDLE().

Referenced by create_interface_sets(), exchange_owned_meshs(), resolve_shared_ents(), moab::ScdInterface::tag_shared_vertices(), and moab::ParallelMergeMesh::TagSharedElements().

◆ create_part()

ErrorCode moab::ParallelComm::create_part ( EntityHandle &  part_out)

Definition at line 8534 of file ParallelComm.cpp.

8535 {
8536  // Mark as invalid so we know that it needs to be updated
8537  globalPartCount = -1;
8538 
8539  // Create set representing part
8540  ErrorCode rval = mbImpl->create_meshset( MESHSET_SET, set_out );
8541  if( MB_SUCCESS != rval ) return rval;
8542 
8543  // Set tag on set
8544  int val = proc_config().proc_rank();
8545  rval = mbImpl->tag_set_data( part_tag(), &set_out, 1, &val );
8546 
8547  if( MB_SUCCESS != rval )
8548  {
8549  mbImpl->delete_entities( &set_out, 1 );
8550  return rval;
8551  }
8552 
8553  if( get_partitioning() )
8554  {
8555  rval = mbImpl->add_entities( get_partitioning(), &set_out, 1 );
8556  if( MB_SUCCESS != rval )
8557  {
8558  mbImpl->delete_entities( &set_out, 1 );
8559  return rval;
8560  }
8561  }
8562 
8563  moab::Range& pSets = this->partition_sets();
8564  if( pSets.index( set_out ) < 0 )
8565  {
8566  pSets.insert( set_out );
8567  }
8568 
8569  return MB_SUCCESS;
8570 }

References moab::Interface::add_entities(), moab::Interface::create_meshset(), moab::Interface::delete_entities(), ErrorCode, get_partitioning(), globalPartCount, moab::Range::index(), moab::Range::insert(), MB_SUCCESS, mbImpl, MESHSET_SET, part_tag(), partition_sets(), proc_config(), moab::ProcConfig::proc_rank(), and moab::Interface::tag_set_data().

◆ define_mpe()

void moab::ParallelComm::define_mpe ( )
private

Definition at line 4366 of file ParallelComm.cpp.

4367 {
4368 #ifdef MOAB_HAVE_MPE
4369  if( myDebug->get_verbosity() == 2 )
4370  {
4371  // Define mpe states used for logging
4372  int success;
4373  MPE_Log_get_state_eventIDs( &IFACE_START, &IFACE_END );
4374  MPE_Log_get_state_eventIDs( &GHOST_START, &GHOST_END );
4375  MPE_Log_get_state_eventIDs( &SHAREDV_START, &SHAREDV_END );
4376  MPE_Log_get_state_eventIDs( &RESOLVE_START, &RESOLVE_END );
4377  MPE_Log_get_state_eventIDs( &ENTITIES_START, &ENTITIES_END );
4378  MPE_Log_get_state_eventIDs( &RHANDLES_START, &RHANDLES_END );
4379  MPE_Log_get_state_eventIDs( &OWNED_START, &OWNED_END );
4380  success = MPE_Describe_state( IFACE_START, IFACE_END, "Resolve interface ents", "green" );
4381  assert( MPE_LOG_OK == success );
4382  success = MPE_Describe_state( GHOST_START, GHOST_END, "Exchange ghost ents", "red" );
4383  assert( MPE_LOG_OK == success );
4384  success = MPE_Describe_state( SHAREDV_START, SHAREDV_END, "Resolve interface vertices", "blue" );
4385  assert( MPE_LOG_OK == success );
4386  success = MPE_Describe_state( RESOLVE_START, RESOLVE_END, "Resolve shared ents", "purple" );
4387  assert( MPE_LOG_OK == success );
4388  success = MPE_Describe_state( ENTITIES_START, ENTITIES_END, "Exchange shared ents", "yellow" );
4389  assert( MPE_LOG_OK == success );
4390  success = MPE_Describe_state( RHANDLES_START, RHANDLES_END, "Remote handles", "cyan" );
4391  assert( MPE_LOG_OK == success );
4392  success = MPE_Describe_state( OWNED_START, OWNED_END, "Exchange owned ents", "black" );
4393  assert( MPE_LOG_OK == success );
4394  }
4395 #endif
4396 }

References moab::DebugOutput::get_verbosity(), MPE_Describe_state, MPE_LOG_OK, and myDebug.

Referenced by resolve_shared_ents().

◆ delete_all_buffers()

void moab::ParallelComm::delete_all_buffers ( )
inlineprivate

reset message buffers to their initial state

delete all buffers, freeing up any memory held by them

Definition at line 1557 of file ParallelComm.hpp.

1558 {
1559  std::vector< Buffer* >::iterator vit;
1560  for( vit = localOwnedBuffs.begin(); vit != localOwnedBuffs.end(); ++vit )
1561  delete( *vit );
1562  localOwnedBuffs.clear();
1563 
1564  for( vit = remoteOwnedBuffs.begin(); vit != remoteOwnedBuffs.end(); ++vit )
1565  delete( *vit );
1566  remoteOwnedBuffs.clear();
1567 }

References localOwnedBuffs, and remoteOwnedBuffs.

Referenced by ~ParallelComm().

◆ delete_entities()

ErrorCode moab::ParallelComm::delete_entities ( Range &  to_delete)

Definition at line 9599 of file ParallelComm.cpp.

9600 {
9601  // Will not look at shared sets yet, but maybe we should
9602  // First, see if any of the entities to delete is shared; then inform the other processors
9603  // about their fate (to be deleted), using a crystal router transfer
9604  ErrorCode rval = MB_SUCCESS;
9605  unsigned char pstat;
9606  EntityHandle tmp_handles[MAX_SHARING_PROCS];
9607  int tmp_procs[MAX_SHARING_PROCS];
9608  unsigned int num_ps;
9609  TupleList ents_to_delete;
9610  ents_to_delete.initialize( 1, 0, 1, 0, to_delete.size() * ( MAX_SHARING_PROCS + 1 ) ); // A little bit of overkill
9611  ents_to_delete.enableWriteAccess();
9612  unsigned int i = 0;
9613  for( Range::iterator it = to_delete.begin(); it != to_delete.end(); ++it )
9614  {
9615  EntityHandle eh = *it; // Entity to be deleted
9616 
9617  rval = get_sharing_data( eh, tmp_procs, tmp_handles, pstat, num_ps );
9618  if( rval != MB_SUCCESS || num_ps == 0 ) continue;
9619  // Add to the tuple list the information to be sent (to the remote procs)
9620  for( unsigned int p = 0; p < num_ps; p++ )
9621  {
9622  ents_to_delete.vi_wr[i] = tmp_procs[p];
9623  ents_to_delete.vul_wr[i] = (unsigned long)tmp_handles[p];
9624  i++;
9625  ents_to_delete.inc_n();
9626  }
9627  }
9628 
9629  gs_data::crystal_data* cd = this->procConfig.crystal_router();
9630  // All communication happens here; no other mpi calls
9631  // Also, this is a collective call
9632  MB_CHK_SET_ERR( cd->gs_transfer( 1, ents_to_delete, 0 ), "Error in tuple transfer" );
9633 
9634  // Add to the range of ents to delete the new ones that were sent from other procs
9635  unsigned int received = ents_to_delete.get_n();
9636  for( i = 0; i < received; i++ )
9637  {
9638  // int from = ents_to_delete.vi_rd[i];
9639  unsigned long valrec = ents_to_delete.vul_rd[i];
9640  to_delete.insert( (EntityHandle)valrec );
9641  }
9642  MB_CHK_SET_ERR( mbImpl->delete_entities( to_delete ), "Error in deleting actual entities" );
9643 
9644  std::set< EntityHandle > good_ents;
9645  for( std::set< EntityHandle >::iterator sst = sharedEnts.begin(); sst != sharedEnts.end(); sst++ )
9646  {
9647  EntityHandle eh = *sst;
9648  int index = to_delete.index( eh );
9649  if( -1 == index ) good_ents.insert( eh );
9650  }
9651  sharedEnts = good_ents;
9652 
9653  // What about shared sets? Who is updating them?
9654  return MB_SUCCESS;
9655 }

References moab::Range::begin(), moab::ProcConfig::crystal_router(), moab::Interface::delete_entities(), moab::TupleList::enableWriteAccess(), moab::Range::end(), ErrorCode, moab::TupleList::get_n(), get_sharing_data(), moab::TupleList::inc_n(), moab::index, moab::Range::index(), moab::TupleList::initialize(), moab::Range::insert(), MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, procConfig, sharedEnts, moab::Range::size(), moab::TupleList::vi_wr, moab::TupleList::vul_rd, and moab::TupleList::vul_wr.

Referenced by moab::NCHelperScrip::create_mesh().

◆ destroy_part()

ErrorCode moab::ParallelComm::destroy_part ( EntityHandle  part)

Definition at line 8572 of file ParallelComm.cpp.

8573 {
8574  // Mark as invalid so we know that it needs to be updated
8575  globalPartCount = -1;
8576 
8577  ErrorCode rval;
8578  if( get_partitioning() )
8579  {
8580  rval = mbImpl->remove_entities( get_partitioning(), &part_id, 1 );
8581  if( MB_SUCCESS != rval ) return rval;
8582  }
8583 
8584  moab::Range& pSets = this->partition_sets();
8585  if( pSets.index( part_id ) >= 0 )
8586  {
8587  pSets.erase( part_id );
8588  }
8589  return mbImpl->delete_entities( &part_id, 1 );
8590 }

References moab::Interface::delete_entities(), moab::Range::erase(), ErrorCode, get_partitioning(), globalPartCount, moab::Range::index(), MB_SUCCESS, mbImpl, partition_sets(), and moab::Interface::remove_entities().

◆ estimate_ents_buffer_size()

int moab::ParallelComm::estimate_ents_buffer_size ( Range &  entities,
const bool  store_remote_handles 
)
private

estimate size required to pack entities

Definition at line 1540 of file ParallelComm.cpp.

1541 {
1542  int buff_size = 0;
1543  std::vector< EntityHandle > dum_connect_vec;
1544  const EntityHandle* connect;
1545  int num_connect;
1546 
1547  int num_verts = entities.num_of_type( MBVERTEX );
1548  // # verts + coords + handles
1549  buff_size += 2 * sizeof( int ) + 3 * sizeof( double ) * num_verts;
1550  if( store_remote_handles ) buff_size += sizeof( EntityHandle ) * num_verts;
1551 
1552  // Do a rough count by looking at first entity of each type
1553  for( EntityType t = MBEDGE; t < MBENTITYSET; t++ )
1554  {
1555  const Range::iterator rit = entities.lower_bound( t );
1556  if( TYPE_FROM_HANDLE( *rit ) != t ) continue;
1557 
1558  ErrorCode result = mbImpl->get_connectivity( *rit, connect, num_connect, false, &dum_connect_vec );
1559  MB_CHK_SET_ERR_RET_VAL( result, "Failed to get connectivity to estimate buffer size", -1 );
1560 
1561  // Number, type, nodes per entity
1562  buff_size += 3 * sizeof( int );
1563  int num_ents = entities.num_of_type( t );
1564  // Connectivity, handle for each ent
1565  buff_size += ( num_connect + 1 ) * sizeof( EntityHandle ) * num_ents;
1566  }
1567 
1568  // Extra entity type at end, passed as int
1569  buff_size += sizeof( int );
1570 
1571  return buff_size;
1572 }

References ErrorCode, moab::Interface::get_connectivity(), moab::Range::lower_bound(), MB_CHK_SET_ERR_RET_VAL, MBEDGE, MBENTITYSET, mbImpl, MBVERTEX, moab::Range::num_of_type(), and moab::TYPE_FROM_HANDLE().

Referenced by pack_entities().

◆ estimate_sets_buffer_size()

int moab::ParallelComm::estimate_sets_buffer_size ( Range &  entities,
const bool  store_remote_handles 
)
private

estimate size required to pack sets

Definition at line 1574 of file ParallelComm.cpp.

1575 {
1576  // Number of sets
1577  int buff_size = sizeof( int );
1578 
1579  // Do a rough count by looking at first entity of each type
1580  Range::iterator rit = entities.lower_bound( MBENTITYSET );
1581  ErrorCode result;
1582 
1583  for( ; rit != entities.end(); ++rit )
1584  {
1585  unsigned int options;
1586  result = mbImpl->get_meshset_options( *rit, options );
1587  MB_CHK_SET_ERR_RET_VAL( result, "Failed to get meshset options", -1 );
1588 
1589  buff_size += sizeof( int );
1590 
1591  Range set_range;
1592  if( options & MESHSET_SET )
1593  {
1594  // Range-based set; count the subranges
1595  result = mbImpl->get_entities_by_handle( *rit, set_range );
1596  MB_CHK_SET_ERR_RET_VAL( result, "Failed to get set entities", -1 );
1597 
1598  // Set range
1599  buff_size += RANGE_SIZE( set_range );
1600  }
1601  else if( options & MESHSET_ORDERED )
1602  {
1603  // Just get the number of entities in the set
1604  int num_ents;
1605  result = mbImpl->get_number_entities_by_handle( *rit, num_ents );
1606  MB_CHK_SET_ERR_RET_VAL( result, "Failed to get number entities in ordered set", -1 );
1607 
1608  // Set vec
1609  buff_size += sizeof( EntityHandle ) * num_ents + sizeof( int );
1610  }
1611 
1612  // Get numbers of parents/children
1613  int num_par, num_ch;
1614  result = mbImpl->num_child_meshsets( *rit, &num_ch );
1615  MB_CHK_SET_ERR_RET_VAL( result, "Failed to get num children", -1 );
1616  result = mbImpl->num_parent_meshsets( *rit, &num_par );
1617  MB_CHK_SET_ERR_RET_VAL( result, "Failed to get num parents", -1 );
1618 
1619  buff_size += ( num_ch + num_par ) * sizeof( EntityHandle ) + 2 * sizeof( int );
1620  }
1621 
1622  return buff_size;
1623 }

References moab::Range::end(), ErrorCode, moab::Interface::get_entities_by_handle(), moab::Interface::get_meshset_options(), moab::Interface::get_number_entities_by_handle(), moab::Range::lower_bound(), MB_CHK_SET_ERR_RET_VAL, MBENTITYSET, mbImpl, MESHSET_SET, moab::Interface::num_child_meshsets(), moab::Interface::num_parent_meshsets(), and moab::RANGE_SIZE().

Referenced by pack_sets().

◆ exchange_all_shared_handles()

ErrorCode moab::ParallelComm::exchange_all_shared_handles ( std::vector< std::vector< SharedEntityData > > &  send_data,
std::vector< std::vector< SharedEntityData > > &  result 
)
private

Every processor sends shared entity handle data to every other processor that it shares entities with. Passed back map is all received data, indexed by processor ID. This function is intended to be used for debugging.

Definition at line 8802 of file ParallelComm.cpp.

8804 {
8805  int ierr;
8806  const int tag = 0;
8807  const MPI_Comm cm = procConfig.proc_comm();
8808  const int num_proc = buffProcs.size();
8809  const std::vector< int > procs( buffProcs.begin(), buffProcs.end() );
8810  std::vector< MPI_Request > recv_req( buffProcs.size(), MPI_REQUEST_NULL );
8811  std::vector< MPI_Request > send_req( buffProcs.size(), MPI_REQUEST_NULL );
8812 
8813  // Set up to receive sizes
8814  std::vector< int > sizes_send( num_proc ), sizes_recv( num_proc );
8815  for( int i = 0; i < num_proc; i++ )
8816  {
8817  ierr = MPI_Irecv( &sizes_recv[i], 1, MPI_INT, procs[i], tag, cm, &recv_req[i] );
8818  if( ierr ) return MB_FILE_WRITE_ERROR;
8819  }
8820 
8821  // Send sizes
8822  assert( num_proc == (int)send_data.size() );
8823 
8824  result.resize( num_proc );
8825  for( int i = 0; i < num_proc; i++ )
8826  {
8827  sizes_send[i] = send_data[i].size();
8828  ierr = MPI_Isend( &sizes_send[i], 1, MPI_INT, buffProcs[i], tag, cm, &send_req[i] );
8829  if( ierr ) return MB_FILE_WRITE_ERROR;
8830  }
8831 
8832  // Receive sizes
8833  std::vector< MPI_Status > stat( num_proc );
8834  ierr = MPI_Waitall( num_proc, recv_req.data(), stat.data() );
8835  if( ierr ) return MB_FILE_WRITE_ERROR;
8836 
8837  // Wait until all sizes are sent (clean up pending req's)
8838  ierr = MPI_Waitall( num_proc, send_req.data(), stat.data() );
8839  if( ierr ) return MB_FILE_WRITE_ERROR;
8840 
8841  // Set up to receive data
8842  for( int i = 0; i < num_proc; i++ )
8843  {
8844  result[i].resize( sizes_recv[i] );
8845  ierr = MPI_Irecv( (void*)( result[i].data() ), sizeof( SharedEntityData ) * sizes_recv[i], MPI_UNSIGNED_CHAR,
8846  buffProcs[i], tag, cm, &recv_req[i] );
8847  if( ierr ) return MB_FILE_WRITE_ERROR;
8848  }
8849 
8850  // Send data
8851  for( int i = 0; i < num_proc; i++ )
8852  {
8853  ierr = MPI_Isend( (void*)( send_data[i].data() ), sizeof( SharedEntityData ) * sizes_send[i], MPI_UNSIGNED_CHAR,
8854  buffProcs[i], tag, cm, &send_req[i] );
8855  if( ierr ) return MB_FILE_WRITE_ERROR;
8856  }
8857 
8858  // Receive data
8859  ierr = MPI_Waitall( num_proc, recv_req.data(), stat.data() );
8860  if( ierr ) return MB_FILE_WRITE_ERROR;
8861 
8862  // Wait until everything is sent to release send buffers
8863  ierr = MPI_Waitall( num_proc, send_req.data(), stat.data() );
8864  if( ierr ) return MB_FILE_WRITE_ERROR;
8865 
8866  return MB_SUCCESS;
8867 }

References buffProcs, MB_FILE_WRITE_ERROR, MB_SUCCESS, moab::ProcConfig::proc_comm(), and procConfig.

Referenced by check_all_shared_handles(), and correct_thin_ghost_layers().

◆ exchange_ghost_cells() [1/2]

ErrorCode moab::ParallelComm::exchange_ghost_cells ( int  ghost_dim,
int  bridge_dim,
int  num_layers,
int  addl_ents,
bool  store_remote_handles,
bool  wait_all = true,
EntityHandle *  file_set = NULL 
)

Exchange ghost cells with neighboring procs Neighboring processors are those sharing an interface with this processor. All entities of dimension ghost_dim within num_layers of interface, measured going through bridge_dim, are exchanged. See MeshTopoUtil::get_bridge_adjacencies for description of bridge adjacencies. If wait_all is false and store_remote_handles is true, MPI_Request objects are available in the sendReqs[2*MAX_SHARING_PROCS] member array, with inactive requests marked as MPI_REQUEST_NULL. If store_remote_handles or wait_all is false, this function returns after all entities have been received and processed.

Parameters
ghost_dimDimension of ghost entities to be exchanged
bridge_dimDimension of entities used to measure layers from interface
num_layersNumber of layers of ghosts requested
addl_entsDimension of additional adjacent entities to exchange with ghosts, 0 if none
store_remote_handlesIf true, send message with new entity handles to source processor
wait_allIf true, function does not return until all send buffers are cleared.

Definition at line 5891 of file ParallelComm.cpp.

5898 {
5899 #ifdef MOAB_HAVE_MPE
5900  if( myDebug->get_verbosity() == 2 )
5901  {
5902  if( !num_layers )
5903  MPE_Log_event( IFACE_START, procConfig.proc_rank(), "Starting interface exchange." );
5904  else
5905  MPE_Log_event( GHOST_START, procConfig.proc_rank(), "Starting ghost exchange." );
5906  }
5907 #endif
5908 
5909  myDebug->tprintf( 1, "Entering exchange_ghost_cells with num_layers = %d\n", num_layers );
5910  if( myDebug->get_verbosity() == 4 )
5911  {
5912  msgs.clear();
5913  msgs.reserve( MAX_SHARING_PROCS );
5914  }
5915 
5916  // If we're only finding out about existing ents, we have to be storing
5917  // remote handles too
5918  assert( num_layers > 0 || store_remote_handles );
5919 
5920  const bool is_iface = !num_layers;
5921 
5922  // Get the b-dimensional interface(s) with with_proc, where b = bridge_dim
5923 
5924  int success;
5925  ErrorCode result = MB_SUCCESS;
5926  int incoming1 = 0, incoming2 = 0;
5927 
5928  reset_all_buffers();
5929 
5930  // When this function is called, buffProcs should already have any
5931  // communicating procs
5932 
5933  //===========================================
5934  // Post ghost irecv's for ghost entities from all communicating procs
5935  //===========================================
5936 #ifdef MOAB_HAVE_MPE
5937  if( myDebug->get_verbosity() == 2 )
5938  {
5939  MPE_Log_event( ENTITIES_START, procConfig.proc_rank(), "Starting entity exchange." );
5940  }
5941 #endif
5942 
5943  // Index reqs the same as buffer/sharing procs indices
5944  std::vector< MPI_Request > recv_ent_reqs( 3 * buffProcs.size(), MPI_REQUEST_NULL ),
5945  recv_remoteh_reqs( 3 * buffProcs.size(), MPI_REQUEST_NULL );
5946  std::vector< unsigned int >::iterator proc_it;
5947  int ind, p;
5948  sendReqs.resize( 3 * buffProcs.size(), MPI_REQUEST_NULL );
5949  for( ind = 0, proc_it = buffProcs.begin(); proc_it != buffProcs.end(); ++proc_it, ind++ )
5950  {
5951  incoming1++;
5952  PRINT_DEBUG_IRECV( procConfig.proc_rank(), buffProcs[ind], remoteOwnedBuffs[ind]->mem_ptr, INITIAL_BUFF_SIZE,
5953  MB_MESG_ENTS_SIZE, incoming1 );
5954  success = MPI_Irecv( remoteOwnedBuffs[ind]->mem_ptr, INITIAL_BUFF_SIZE, MPI_UNSIGNED_CHAR, buffProcs[ind],
5955  MB_MESG_ENTS_SIZE, procConfig.proc_comm(), &recv_ent_reqs[3 * ind] );
5956  if( success != MPI_SUCCESS )
5957  {
5958  MB_SET_ERR( MB_FAILURE, "Failed to post irecv in ghost exchange" );
5959  }
5960  }
5961 
5962  //===========================================
5963  // Get entities to be sent to neighbors
5964  //===========================================
5965  Range sent_ents[MAX_SHARING_PROCS], allsent, tmp_range;
5966  TupleList entprocs;
5967  int dum_ack_buff;
5968  result = get_sent_ents( is_iface, bridge_dim, ghost_dim, num_layers, addl_ents, sent_ents, allsent, entprocs );
5969  MB_CHK_SET_ERR( result, "get_sent_ents failed" );
5970 
5971  // augment file set with the entities to be sent
5972  // we might have created new entities if addl_ents>0, edges and/or faces
5973  if( addl_ents > 0 && file_set && !allsent.empty() )
5974  {
5975  result = mbImpl->add_entities( *file_set, allsent );
5976  MB_CHK_SET_ERR( result, "Failed to add new sub-entities to set" );
5977  }
5978  myDebug->tprintf( 1, "allsent ents compactness (size) = %f (%lu)\n", allsent.compactness(),
5979  (unsigned long)allsent.size() );
5980 
5981  //===========================================
5982  // Pack and send ents from this proc to others
5983  //===========================================
5984  for( p = 0, proc_it = buffProcs.begin(); proc_it != buffProcs.end(); ++proc_it, p++ )
5985  {
5986  myDebug->tprintf( 1, "Sent ents compactness (size) = %f (%lu)\n", sent_ents[p].compactness(),
5987  (unsigned long)sent_ents[p].size() );
5988 
5989  // Reserve space on front for size and for initial buff size
5990  localOwnedBuffs[p]->reset_buffer( sizeof( int ) );
5991 
5992  // Entities
5993  result = pack_entities( sent_ents[p], localOwnedBuffs[p], store_remote_handles, buffProcs[p], is_iface,
5994  &entprocs, &allsent );
5995  MB_CHK_SET_ERR( result, "Packing entities failed" );
5996 
5997  if( myDebug->get_verbosity() == 4 )
5998  {
5999  msgs.resize( msgs.size() + 1 );
6000  msgs.back() = new Buffer( *localOwnedBuffs[p] );
6001  }
6002 
6003  // Send the buffer (size stored in front in send_buffer)
6004  result = send_buffer( *proc_it, localOwnedBuffs[p], MB_MESG_ENTS_SIZE, sendReqs[3 * p],
6005  recv_ent_reqs[3 * p + 2], &dum_ack_buff, incoming1, MB_MESG_REMOTEH_SIZE,
6006  ( !is_iface && store_remote_handles ? // this used for ghosting only
6007  localOwnedBuffs[p]
6008  : NULL ),
6009  &recv_remoteh_reqs[3 * p], &incoming2 );
6010  MB_CHK_SET_ERR( result, "Failed to Isend in ghost exchange" );
6011  }
6012 
6013  entprocs.reset();
6014 
6015  //===========================================
6016  // Receive/unpack new entities
6017  //===========================================
6018  // Number of incoming messages for ghosts is the number of procs we
6019  // communicate with; for iface, it's the number of those with lower rank
6020  MPI_Status status;
6021  std::vector< std::vector< EntityHandle > > recd_ents( buffProcs.size() );
6022  std::vector< std::vector< EntityHandle > > L1hloc( buffProcs.size() ), L1hrem( buffProcs.size() );
6023  std::vector< std::vector< int > > L1p( buffProcs.size() );
6024  std::vector< EntityHandle > L2hloc, L2hrem;
6025  std::vector< unsigned int > L2p;
6026  std::vector< EntityHandle > new_ents;
6027 
6028  while( incoming1 )
6029  {
6030  // Wait for all recvs of ghost ents before proceeding to sending remote handles,
6031  // b/c some procs may have sent to a 3rd proc ents owned by me;
6032  PRINT_DEBUG_WAITANY( recv_ent_reqs, MB_MESG_ENTS_SIZE, procConfig.proc_rank() );
6033 
6034  success = MPI_Waitany( 3 * buffProcs.size(), recv_ent_reqs.data(), &ind, &status );
6035  if( MPI_SUCCESS != success )
6036  {
6037  MB_SET_ERR( MB_FAILURE, "Failed in waitany in ghost exchange" );
6038  }
6039 
6040  PRINT_DEBUG_RECD( status );
6041 
6042  // OK, received something; decrement incoming counter
6043  incoming1--;
6044  bool done = false;
6045 
6046  // In case ind is for ack, we need index of one before it
6047  unsigned int base_ind = 3 * ( ind / 3 );
6048  result = recv_buffer( MB_MESG_ENTS_SIZE, status, remoteOwnedBuffs[ind / 3], recv_ent_reqs[base_ind + 1],
6049  recv_ent_reqs[base_ind + 2], incoming1, localOwnedBuffs[ind / 3], sendReqs[base_ind + 1],
6050  sendReqs[base_ind + 2], done,
6051  ( !is_iface && store_remote_handles ? localOwnedBuffs[ind / 3] : NULL ),
6052  MB_MESG_REMOTEH_SIZE, // maybe base_ind+1?
6053  &recv_remoteh_reqs[base_ind + 1], &incoming2 );
6054  MB_CHK_SET_ERR( result, "Failed to receive buffer" );
6055 
6056  if( done )
6057  {
6058  if( myDebug->get_verbosity() == 4 )
6059  {
6060  msgs.resize( msgs.size() + 1 );
6061  msgs.back() = new Buffer( *remoteOwnedBuffs[ind / 3] );
6062  }
6063 
6064  // Message completely received - process buffer that was sent
6065  remoteOwnedBuffs[ind / 3]->reset_ptr( sizeof( int ) );
6066  result = unpack_entities( remoteOwnedBuffs[ind / 3]->buff_ptr, store_remote_handles, ind / 3, is_iface,
6067  L1hloc, L1hrem, L1p, L2hloc, L2hrem, L2p, new_ents );
6068  if( MB_SUCCESS != result )
6069  {
6070  std::cout << "Failed to unpack entities. Buffer contents:" << std::endl;
6071  print_buffer( remoteOwnedBuffs[ind / 3]->mem_ptr, MB_MESG_ENTS_SIZE, buffProcs[ind / 3], false );
6072  return result;
6073  }
6074 
6075  if( recv_ent_reqs.size() != 3 * buffProcs.size() )
6076  {
6077  // Post irecv's for remote handles from new proc; shouldn't be iface,
6078  // since we know about all procs we share with
6079  assert( !is_iface );
6080  recv_remoteh_reqs.resize( 3 * buffProcs.size(), MPI_REQUEST_NULL );
6081  for( unsigned int i = recv_ent_reqs.size(); i < 3 * buffProcs.size(); i += 3 )
6082  {
6083  localOwnedBuffs[i / 3]->reset_buffer();
6084  incoming2++;
6085  PRINT_DEBUG_IRECV( procConfig.proc_rank(), buffProcs[i / 3], localOwnedBuffs[i / 3]->mem_ptr,
6086  INITIAL_BUFF_SIZE, MB_MESG_REMOTEH_SIZE, incoming2 );
6087  success = MPI_Irecv( localOwnedBuffs[i / 3]->mem_ptr, INITIAL_BUFF_SIZE, MPI_UNSIGNED_CHAR,
6088  buffProcs[i / 3], MB_MESG_REMOTEH_SIZE, procConfig.proc_comm(),
6089  &recv_remoteh_reqs[i] );
6090  if( success != MPI_SUCCESS )
6091  {
6092  MB_SET_ERR( MB_FAILURE, "Failed to post irecv for remote handles in ghost exchange" );
6093  }
6094  }
6095  recv_ent_reqs.resize( 3 * buffProcs.size(), MPI_REQUEST_NULL );
6096  sendReqs.resize( 3 * buffProcs.size(), MPI_REQUEST_NULL );
6097  }
6098  }
6099  }
6100 
6101  // Add requests for any new addl procs
6102  if( recv_ent_reqs.size() != 3 * buffProcs.size() )
6103  {
6104  // Shouldn't get here...
6105  MB_SET_ERR( MB_FAILURE, "Requests length doesn't match proc count in ghost exchange" );
6106  }
6107 
6108 #ifdef MOAB_HAVE_MPE
6109  if( myDebug->get_verbosity() == 2 )
6110  {
6111  MPE_Log_event( ENTITIES_END, procConfig.proc_rank(), "Ending entity exchange." );
6112  }
6113 #endif
6114 
6115  if( is_iface )
6116  {
6117  // Need to check over entities I sent and make sure I received
6118  // handles for them from all expected procs; if not, need to clean
6119  // them up
6120  result = check_clean_iface( allsent );
6121  if( MB_SUCCESS != result ) std::cout << "Failed check." << std::endl;
6122 
6123  // Now set the shared/interface tag on non-vertex entities on interface
6124  result = tag_iface_entities();
6125  MB_CHK_SET_ERR( result, "Failed to tag iface entities" );
6126 
6127 #ifndef NDEBUG
6128  result = check_sent_ents( allsent );
6129  if( MB_SUCCESS != result ) std::cout << "Failed check." << std::endl;
6130  result = check_all_shared_handles( true );
6131  if( MB_SUCCESS != result ) std::cout << "Failed check." << std::endl;
6132 #endif
6133 
6134 #ifdef MOAB_HAVE_MPE
6135  if( myDebug->get_verbosity() == 2 )
6136  {
6137  MPE_Log_event( IFACE_END, procConfig.proc_rank(), "Ending interface exchange." );
6138  }
6139 #endif
6140 
6141  //===========================================
6142  // Wait if requested
6143  //===========================================
6144  if( wait_all )
6145  {
6146  if( myDebug->get_verbosity() == 5 )
6147  {
6148  success = MPI_Barrier( procConfig.proc_comm() );
6149  }
6150  else
6151  {
6152  MPI_Status mult_status[3 * MAX_SHARING_PROCS];
6153  success = MPI_Waitall( 3 * buffProcs.size(), recv_ent_reqs.data(), mult_status );
6154  if( MPI_SUCCESS != success )
6155  {
6156  MB_SET_ERR( MB_FAILURE, "Failed in waitall in ghost exchange" );
6157  }
6158  success = MPI_Waitall( 3 * buffProcs.size(), sendReqs.data(), mult_status );
6159  if( MPI_SUCCESS != success )
6160  {
6161  MB_SET_ERR( MB_FAILURE, "Failed in waitall in ghost exchange" );
6162  }
6163  /*success = MPI_Waitall(3*buffProcs.size(), &recv_remoteh_reqs[0], mult_status);
6164  if (MPI_SUCCESS != success) {
6165  MB_SET_ERR(MB_FAILURE, "Failed in waitall in ghost exchange");
6166  }*/
6167  }
6168  }
6169 
6170  myDebug->tprintf( 1, "Total number of shared entities = %lu.\n", (unsigned long)sharedEnts.size() );
6171  myDebug->tprintf( 1, "Exiting exchange_ghost_cells for is_iface==true \n" );
6172 
6173  return MB_SUCCESS;
6174  }
6175 
6176  // we still need to wait on sendReqs, if they are not fulfilled yet
6177  if( wait_all )
6178  {
6179  if( myDebug->get_verbosity() == 5 )
6180  {
6181  success = MPI_Barrier( procConfig.proc_comm() );
6182  }
6183  else
6184  {
6185  MPI_Status mult_status[3 * MAX_SHARING_PROCS];
6186  success = MPI_Waitall( 3 * buffProcs.size(), sendReqs.data(), mult_status );
6187  if( MPI_SUCCESS != success )
6188  {
6189  MB_SET_ERR( MB_FAILURE, "Failed in waitall in ghost exchange" );
6190  }
6191  }
6192  }
6193  //===========================================
6194  // Send local handles for new ghosts to owner, then add
6195  // those to ghost list for that owner
6196  //===========================================
6197  for( p = 0, proc_it = buffProcs.begin(); proc_it != buffProcs.end(); ++proc_it, p++ )
6198  {
6199 
6200  // Reserve space on front for size and for initial buff size
6201  remoteOwnedBuffs[p]->reset_buffer( sizeof( int ) );
6202 
6203  result = pack_remote_handles( L1hloc[p], L1hrem[p], L1p[p], *proc_it, remoteOwnedBuffs[p] );
6204  MB_CHK_SET_ERR( result, "Failed to pack remote handles" );
6205  remoteOwnedBuffs[p]->set_stored_size();
6206 
6207  if( myDebug->get_verbosity() == 4 )
6208  {
6209  msgs.resize( msgs.size() + 1 );
6210  msgs.back() = new Buffer( *remoteOwnedBuffs[p] );
6211  }
6212  result = send_buffer( buffProcs[p], remoteOwnedBuffs[p], MB_MESG_REMOTEH_SIZE, sendReqs[3 * p],
6213  recv_remoteh_reqs[3 * p + 2], &dum_ack_buff, incoming2 );
6214  MB_CHK_SET_ERR( result, "Failed to send remote handles" );
6215  }
6216 
6217  //===========================================
6218  // Process remote handles of my ghosteds
6219  //===========================================
6220  while( incoming2 )
6221  {
6222  PRINT_DEBUG_WAITANY( recv_remoteh_reqs, MB_MESG_REMOTEH_SIZE, procConfig.proc_rank() );
6223  success = MPI_Waitany( 3 * buffProcs.size(), recv_remoteh_reqs.data(), &ind, &status );
6224  if( MPI_SUCCESS != success )
6225  {
6226  MB_SET_ERR( MB_FAILURE, "Failed in waitany in ghost exchange" );
6227  }
6228 
6229  // OK, received something; decrement incoming counter
6230  incoming2--;
6231 
6232  PRINT_DEBUG_RECD( status );
6233 
6234  bool done = false;
6235  unsigned int base_ind = 3 * ( ind / 3 );
6236  result = recv_buffer( MB_MESG_REMOTEH_SIZE, status, localOwnedBuffs[ind / 3], recv_remoteh_reqs[base_ind + 1],
6237  recv_remoteh_reqs[base_ind + 2], incoming2, remoteOwnedBuffs[ind / 3],
6238  sendReqs[base_ind + 1], sendReqs[base_ind + 2], done );
6239  MB_CHK_SET_ERR( result, "Failed to receive remote handles" );
6240  if( done )
6241  {
6242  // Incoming remote handles
6243  if( myDebug->get_verbosity() == 4 )
6244  {
6245  msgs.resize( msgs.size() + 1 );
6246  msgs.back() = new Buffer( *localOwnedBuffs[ind / 3] );
6247  }
6248  localOwnedBuffs[ind / 3]->reset_ptr( sizeof( int ) );
6249  result =
6250  unpack_remote_handles( buffProcs[ind / 3], localOwnedBuffs[ind / 3]->buff_ptr, L2hloc, L2hrem, L2p );
6251  MB_CHK_SET_ERR( result, "Failed to unpack remote handles" );
6252  }
6253  }
6254 
6255 #ifdef MOAB_HAVE_MPE
6256  if( myDebug->get_verbosity() == 2 )
6257  {
6258  MPE_Log_event( RHANDLES_END, procConfig.proc_rank(), "Ending remote handles." );
6259  MPE_Log_event( GHOST_END, procConfig.proc_rank(), "Ending ghost exchange (still doing checks)." );
6260  }
6261 #endif
6262 
6263  //===========================================
6264  // Wait if requested
6265  //===========================================
6266  if( wait_all )
6267  {
6268  if( myDebug->get_verbosity() == 5 )
6269  {
6270  success = MPI_Barrier( procConfig.proc_comm() );
6271  }
6272  else
6273  {
6274  MPI_Status mult_status[3 * MAX_SHARING_PROCS];
6275  success = MPI_Waitall( 3 * buffProcs.size(), recv_remoteh_reqs.data(), mult_status );
6276  if( MPI_SUCCESS == success ) success = MPI_Waitall( 3 * buffProcs.size(), sendReqs.data(), mult_status );
6277  }
6278  if( MPI_SUCCESS != success )
6279  {
6280  MB_SET_ERR( MB_FAILURE, "Failed in waitall in ghost exchange" );
6281  }
6282  }
6283 
6284 #ifndef NDEBUG
6285  result = check_sent_ents( allsent );
6286  MB_CHK_SET_ERR( result, "Failed check on shared entities" );
6287  result = check_all_shared_handles( true );
6288  MB_CHK_SET_ERR( result, "Failed check on all shared handles" );
6289 #endif
6290 
6291  if( file_set && !new_ents.empty() )
6292  {
6293  result = mbImpl->add_entities( *file_set, new_ents.data(), new_ents.size() );
6294  MB_CHK_SET_ERR( result, "Failed to add new entities to set" );
6295  }
6296 
6297  myDebug->tprintf( 1, "Total number of shared entities = %lu.\n", (unsigned long)sharedEnts.size() );
6298  myDebug->tprintf( 1, "Exiting exchange_ghost_cells for is_iface==false \n" );
6299 
6300  return MB_SUCCESS;
6301 }

References moab::Interface::add_entities(), buffProcs, check_all_shared_handles(), check_clean_iface(), check_sent_ents(), moab::Range::compactness(), moab::Range::empty(), ErrorCode, get_sent_ents(), moab::DebugOutput::get_verbosity(), INITIAL_BUFF_SIZE, localOwnedBuffs, MAX_SHARING_PROCS, MB_CHK_SET_ERR, moab::MB_MESG_ENTS_SIZE, moab::MB_MESG_REMOTEH_SIZE, MB_SET_ERR, MB_SUCCESS, mbImpl, MPE_Log_event, moab::msgs, myDebug, pack_entities(), pack_remote_handles(), print_buffer(), PRINT_DEBUG_IRECV, PRINT_DEBUG_RECD, PRINT_DEBUG_WAITANY, moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_rank(), procConfig, recv_buffer(), remoteOwnedBuffs, moab::TupleList::reset(), reset_all_buffers(), send_buffer(), sendReqs, sharedEnts, moab::Range::size(), size(), tag_iface_entities(), moab::DebugOutput::tprintf(), unpack_entities(), and unpack_remote_handles().

Referenced by moab::NestedRefine::exchange_ghosts(), iMOAB_DetermineGhostEntities(), moab::ReadParallel::load_file(), resolve_shared_ents(), and moab::ParallelMergeMesh::TagSharedElements().

◆ exchange_ghost_cells() [2/2]

ErrorCode moab::ParallelComm::exchange_ghost_cells ( ParallelComm **  pc,
unsigned int  num_procs,
int  ghost_dim,
int  bridge_dim,
int  num_layers,
int  addl_ents,
bool  store_remote_handles,
EntityHandle *  file_sets = NULL 
)
static

Static version of exchange_ghost_cells, exchanging info through buffers rather than messages.

Definition at line 6832 of file ParallelComm.cpp.

6840 {
6841  // Static version of function, exchanging info through buffers rather
6842  // than through messages
6843 
6844  // If we're only finding out about existing ents, we have to be storing
6845  // remote handles too
6846  assert( num_layers > 0 || store_remote_handles );
6847 
6848  const bool is_iface = !num_layers;
6849 
6850  unsigned int ind;
6851  ParallelComm* pc;
6852  ErrorCode result = MB_SUCCESS;
6853 
6854  std::vector< Error* > ehs( num_procs );
6855  for( unsigned int i = 0; i < num_procs; i++ )
6856  {
6857  result = pcs[i]->get_moab()->query_interface( ehs[i] );
6858  assert( MB_SUCCESS == result );
6859  }
6860 
6861  // When this function is called, buffProcs should already have any
6862  // communicating procs
6863 
6864  //===========================================
6865  // Get entities to be sent to neighbors
6866  //===========================================
6867 
6868  // Done in a separate loop over procs because sometimes later procs
6869  // need to add info to earlier procs' messages
6870  Range sent_ents[MAX_SHARING_PROCS][MAX_SHARING_PROCS], allsent[MAX_SHARING_PROCS];
6871 
6872  //===========================================
6873  // Get entities to be sent to neighbors
6874  //===========================================
6875  TupleList entprocs[MAX_SHARING_PROCS];
6876  for( unsigned int p = 0; p < num_procs; p++ )
6877  {
6878  pc = pcs[p];
6879  result = pc->get_sent_ents( is_iface, bridge_dim, ghost_dim, num_layers, addl_ents, sent_ents[p], allsent[p],
6880  entprocs[p] );
6881  MB_CHK_SET_ERR( result, "p = " << p << ", get_sent_ents failed" );
6882 
6883  //===========================================
6884  // Pack entities into buffers
6885  //===========================================
6886  for( ind = 0; ind < pc->buffProcs.size(); ind++ )
6887  {
6888  // Entities
6889  pc->localOwnedBuffs[ind]->reset_ptr( sizeof( int ) );
6890  result = pc->pack_entities( sent_ents[p][ind], pc->localOwnedBuffs[ind], store_remote_handles,
6891  pc->buffProcs[ind], is_iface, &entprocs[p], &allsent[p] );
6892  MB_CHK_SET_ERR( result, "p = " << p << ", packing entities failed" );
6893  }
6894 
6895  entprocs[p].reset();
6896  }
6897 
6898  //===========================================
6899  // Receive/unpack new entities
6900  //===========================================
6901  // Number of incoming messages for ghosts is the number of procs we
6902  // communicate with; for iface, it's the number of those with lower rank
6903  std::vector< std::vector< EntityHandle > > L1hloc[MAX_SHARING_PROCS], L1hrem[MAX_SHARING_PROCS];
6904  std::vector< std::vector< int > > L1p[MAX_SHARING_PROCS];
6905  std::vector< EntityHandle > L2hloc[MAX_SHARING_PROCS], L2hrem[MAX_SHARING_PROCS];
6906  std::vector< unsigned int > L2p[MAX_SHARING_PROCS];
6907  std::vector< EntityHandle > new_ents[MAX_SHARING_PROCS];
6908 
6909  for( unsigned int p = 0; p < num_procs; p++ )
6910  {
6911  L1hloc[p].resize( pcs[p]->buffProcs.size() );
6912  L1hrem[p].resize( pcs[p]->buffProcs.size() );
6913  L1p[p].resize( pcs[p]->buffProcs.size() );
6914  }
6915 
6916  for( unsigned int p = 0; p < num_procs; p++ )
6917  {
6918  pc = pcs[p];
6919 
6920  for( ind = 0; ind < pc->buffProcs.size(); ind++ )
6921  {
6922  // Incoming ghost entities; unpack; returns entities received
6923  // both from sending proc and from owning proc (which may be different)
6924 
6925  // Buffer could be empty, which means there isn't any message to
6926  // unpack (due to this comm proc getting added as a result of indirect
6927  // communication); just skip this unpack
6928  if( pc->localOwnedBuffs[ind]->get_stored_size() == 0 ) continue;
6929 
6930  unsigned int to_p = pc->buffProcs[ind];
6931  pc->localOwnedBuffs[ind]->reset_ptr( sizeof( int ) );
6932  result = pcs[to_p]->unpack_entities( pc->localOwnedBuffs[ind]->buff_ptr, store_remote_handles, ind,
6933  is_iface, L1hloc[to_p], L1hrem[to_p], L1p[to_p], L2hloc[to_p],
6934  L2hrem[to_p], L2p[to_p], new_ents[to_p] );
6935  MB_CHK_SET_ERR( result, "p = " << p << ", failed to unpack entities" );
6936  }
6937  }
6938 
6939  if( is_iface )
6940  {
6941  // Need to check over entities I sent and make sure I received
6942  // handles for them from all expected procs; if not, need to clean
6943  // them up
6944  for( unsigned int p = 0; p < num_procs; p++ )
6945  {
6946  result = pcs[p]->check_clean_iface( allsent[p] );
6947  MB_CHK_SET_ERR( result, "p = " << p << ", failed to check on shared entities" );
6948  }
6949 
6950 #ifndef NDEBUG
6951  for( unsigned int p = 0; p < num_procs; p++ )
6952  {
6953  result = pcs[p]->check_sent_ents( allsent[p] );
6954  MB_CHK_SET_ERR( result, "p = " << p << ", failed to check on shared entities" );
6955  }
6956  result = check_all_shared_handles( pcs, num_procs );
6957  MB_CHK_SET_ERR( result, "Failed to check on all shared handles" );
6958 #endif
6959  return MB_SUCCESS;
6960  }
6961 
6962  //===========================================
6963  // Send local handles for new ghosts to owner, then add
6964  // those to ghost list for that owner
6965  //===========================================
6966  std::vector< unsigned int >::iterator proc_it;
6967  for( unsigned int p = 0; p < num_procs; p++ )
6968  {
6969  pc = pcs[p];
6970 
6971  for( ind = 0, proc_it = pc->buffProcs.begin(); proc_it != pc->buffProcs.end(); ++proc_it, ind++ )
6972  {
6973  // Skip if iface layer and higher-rank proc
6974  pc->localOwnedBuffs[ind]->reset_ptr( sizeof( int ) );
6975  result = pc->pack_remote_handles( L1hloc[p][ind], L1hrem[p][ind], L1p[p][ind], *proc_it,
6976  pc->localOwnedBuffs[ind] );
6977  MB_CHK_SET_ERR( result, "p = " << p << ", failed to pack remote handles" );
6978  }
6979  }
6980 
6981  //===========================================
6982  // Process remote handles of my ghosteds
6983  //===========================================
6984  for( unsigned int p = 0; p < num_procs; p++ )
6985  {
6986  pc = pcs[p];
6987 
6988  for( ind = 0, proc_it = pc->buffProcs.begin(); proc_it != pc->buffProcs.end(); ++proc_it, ind++ )
6989  {
6990  // Incoming remote handles
6991  unsigned int to_p = pc->buffProcs[ind];
6992  pc->localOwnedBuffs[ind]->reset_ptr( sizeof( int ) );
6993  result = pcs[to_p]->unpack_remote_handles( p, pc->localOwnedBuffs[ind]->buff_ptr, L2hloc[to_p],
6994  L2hrem[to_p], L2p[to_p] );
6995  MB_CHK_SET_ERR( result, "p = " << p << ", failed to unpack remote handles" );
6996  }
6997  }
6998 
6999 #ifndef NDEBUG
7000  for( unsigned int p = 0; p < num_procs; p++ )
7001  {
7002  result = pcs[p]->check_sent_ents( allsent[p] );
7003  MB_CHK_SET_ERR( result, "p = " << p << ", failed to check on shared entities" );
7004  }
7005 
7006  result = ParallelComm::check_all_shared_handles( pcs, num_procs );
7007  MB_CHK_SET_ERR( result, "Failed to check on all shared handles" );
7008 #endif
7009 
7010  if( file_sets )
7011  {
7012  for( unsigned int p = 0; p < num_procs; p++ )
7013  {
7014  if( new_ents[p].empty() ) continue;
7015  result = pcs[p]->get_moab()->add_entities( file_sets[p], new_ents[p].data(), new_ents[p].size() );
7016  MB_CHK_SET_ERR( result, "p = " << p << ", failed to add new entities to set" );
7017  }
7018  }
7019 
7020  return MB_SUCCESS;
7021 }

References moab::Interface::add_entities(), buffProcs, check_all_shared_handles(), check_clean_iface(), check_sent_ents(), ErrorCode, get_moab(), get_sent_ents(), localOwnedBuffs, MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SUCCESS, pack_entities(), pack_remote_handles(), moab::Interface::query_interface(), moab::TupleList::reset(), size(), unpack_entities(), and unpack_remote_handles().

◆ exchange_owned_mesh()

ErrorCode moab::ParallelComm::exchange_owned_mesh ( std::vector< unsigned int > &  exchange_procs,
std::vector< Range * > &  exchange_ents,
std::vector< MPI_Request > &  recv_ent_reqs,
std::vector< MPI_Request > &  recv_remoteh_reqs,
const bool  recv_posted,
bool  store_remote_handles,
bool  wait_all,
bool  migrate = false 
)

Exchange owned mesh for input mesh entities and sets This function is called twice by exchange_owned_meshs to exchange entities before sets.

Parameters
migrateif the owner if entities are changed or not

Definition at line 7172 of file ParallelComm.cpp.

7180 {
7181 #ifdef MOAB_HAVE_MPE
7182  if( myDebug->get_verbosity() == 2 )
7183  {
7184  MPE_Log_event( OWNED_START, procConfig.proc_rank(), "Starting owned ents exchange." );
7185  }
7186 #endif
7187 
7188  myDebug->tprintf( 1, "Entering exchange_owned_mesh\n" );
7189  if( myDebug->get_verbosity() == 4 )
7190  {
7191  msgs.clear();
7192  msgs.reserve( MAX_SHARING_PROCS );
7193  }
7194  unsigned int i;
7195  int ind, success;
7196  ErrorCode result = MB_SUCCESS;
7197  int incoming1 = 0, incoming2 = 0;
7198 
7199  // Set buffProcs with communicating procs
7200  unsigned int n_proc = exchange_procs.size();
7201  for( i = 0; i < n_proc; i++ )
7202  {
7203  ind = get_buffers( exchange_procs[i] );
7204  result = add_verts( *exchange_ents[i] );
7205  MB_CHK_SET_ERR( result, "Failed to add verts" );
7206 
7207  // Filter out entities already shared with destination
7208  Range tmp_range;
7209  result = filter_pstatus( *exchange_ents[i], PSTATUS_SHARED, PSTATUS_AND, buffProcs[ind], &tmp_range );
7210  MB_CHK_SET_ERR( result, "Failed to filter on owner" );
7211  if( !tmp_range.empty() )
7212  {
7213  *exchange_ents[i] = subtract( *exchange_ents[i], tmp_range );
7214  }
7215  }
7216 
7217  //===========================================
7218  // Post ghost irecv's for entities from all communicating procs
7219  //===========================================
7220 #ifdef MOAB_HAVE_MPE
7221  if( myDebug->get_verbosity() == 2 )
7222  {
7223  MPE_Log_event( ENTITIES_START, procConfig.proc_rank(), "Starting entity exchange." );
7224  }
7225 #endif
7226 
7227  // Index reqs the same as buffer/sharing procs indices
7228  if( !recv_posted )
7229  {
7230  reset_all_buffers();
7231  recv_ent_reqs.resize( 3 * buffProcs.size(), MPI_REQUEST_NULL );
7232  recv_remoteh_reqs.resize( 3 * buffProcs.size(), MPI_REQUEST_NULL );
7233  sendReqs.resize( 3 * buffProcs.size(), MPI_REQUEST_NULL );
7234 
7235  for( i = 0; i < n_proc; i++ )
7236  {
7237  ind = get_buffers( exchange_procs[i] );
7238  incoming1++;
7239  PRINT_DEBUG_IRECV( procConfig.proc_rank(), buffProcs[ind], remoteOwnedBuffs[ind]->mem_ptr,
7240  INITIAL_BUFF_SIZE, MB_MESG_ENTS_SIZE, incoming1 );
7241  success = MPI_Irecv( remoteOwnedBuffs[ind]->mem_ptr, INITIAL_BUFF_SIZE, MPI_UNSIGNED_CHAR, buffProcs[ind],
7242  MB_MESG_ENTS_SIZE, procConfig.proc_comm(), &recv_ent_reqs[3 * ind] );
7243  if( success != MPI_SUCCESS )
7244  {
7245  MB_SET_ERR( MB_FAILURE, "Failed to post irecv in owned entity exchange" );
7246  }
7247  }
7248  }
7249  else
7250  incoming1 += n_proc;
7251 
7252  //===========================================
7253  // Get entities to be sent to neighbors
7254  // Need to get procs each entity is sent to
7255  //===========================================
7256  Range allsent, tmp_range;
7257  int dum_ack_buff;
7258  int npairs = 0;
7259  TupleList entprocs;
7260  for( i = 0; i < n_proc; i++ )
7261  {
7262  int n_ents = exchange_ents[i]->size();
7263  if( n_ents > 0 )
7264  {
7265  npairs += n_ents; // Get the total # of proc/handle pairs
7266  allsent.merge( *exchange_ents[i] );
7267  }
7268  }
7269 
7270  // Allocate a TupleList of that size
7271  entprocs.initialize( 1, 0, 1, 0, npairs );
7272  entprocs.enableWriteAccess();
7273 
7274  // Put the proc/handle pairs in the list
7275  for( i = 0; i < n_proc; i++ )
7276  {
7277  for( Range::iterator rit = exchange_ents[i]->begin(); rit != exchange_ents[i]->end(); ++rit )
7278  {
7279  entprocs.vi_wr[entprocs.get_n()] = exchange_procs[i];
7280  entprocs.vul_wr[entprocs.get_n()] = *rit;
7281  entprocs.inc_n();
7282  }
7283  }
7284 
7285  // Sort by handle
7286  moab::TupleList::buffer sort_buffer;
7287  sort_buffer.buffer_init( npairs );
7288  entprocs.sort( 1, &sort_buffer );
7289  sort_buffer.reset();
7290 
7291  myDebug->tprintf( 1, "allsent ents compactness (size) = %f (%lu)\n", allsent.compactness(),
7292  (unsigned long)allsent.size() );
7293 
7294  //===========================================
7295  // Pack and send ents from this proc to others
7296  //===========================================
7297  for( i = 0; i < n_proc; i++ )
7298  {
7299  ind = get_buffers( exchange_procs[i] );
7300  myDebug->tprintf( 1, "Sent ents compactness (size) = %f (%lu)\n", exchange_ents[i]->compactness(),
7301  (unsigned long)exchange_ents[i]->size() );
7302  // Reserve space on front for size and for initial buff size
7303  localOwnedBuffs[ind]->reset_buffer( sizeof( int ) );
7304  result = pack_buffer( *exchange_ents[i], false, true, store_remote_handles, buffProcs[ind],
7305  localOwnedBuffs[ind], &entprocs, &allsent );
7306 
7307  if( myDebug->get_verbosity() == 4 )
7308  {
7309  msgs.resize( msgs.size() + 1 );
7310  msgs.back() = new Buffer( *localOwnedBuffs[ind] );
7311  }
7312 
7313  // Send the buffer (size stored in front in send_buffer)
7314  result = send_buffer( exchange_procs[i], localOwnedBuffs[ind], MB_MESG_ENTS_SIZE, sendReqs[3 * ind],
7315  recv_ent_reqs[3 * ind + 2], &dum_ack_buff, incoming1, MB_MESG_REMOTEH_SIZE,
7316  ( store_remote_handles ? localOwnedBuffs[ind] : NULL ), &recv_remoteh_reqs[3 * ind],
7317  &incoming2 );
7318  MB_CHK_SET_ERR( result, "Failed to Isend in ghost exchange" );
7319  }
7320 
7321  entprocs.reset();
7322 
7323  //===========================================
7324  // Receive/unpack new entities
7325  //===========================================
7326  // Number of incoming messages is the number of procs we communicate with
7327  MPI_Status status;
7328  std::vector< std::vector< EntityHandle > > recd_ents( buffProcs.size() );
7329  std::vector< std::vector< EntityHandle > > L1hloc( buffProcs.size() ), L1hrem( buffProcs.size() );
7330  std::vector< std::vector< int > > L1p( buffProcs.size() );
7331  std::vector< EntityHandle > L2hloc, L2hrem;
7332  std::vector< unsigned int > L2p;
7333  std::vector< EntityHandle > new_ents;
7334 
7335  while( incoming1 )
7336  {
7337  // Wait for all recvs of ents before proceeding to sending remote handles,
7338  // b/c some procs may have sent to a 3rd proc ents owned by me;
7339  PRINT_DEBUG_WAITANY( recv_ent_reqs, MB_MESG_ENTS_SIZE, procConfig.proc_rank() );
7340 
7341  success = MPI_Waitany( 3 * buffProcs.size(), recv_ent_reqs.data(), &ind, &status );
7342  if( MPI_SUCCESS != success )
7343  {
7344  MB_SET_ERR( MB_FAILURE, "Failed in waitany in owned entity exchange" );
7345  }
7346 
7347  PRINT_DEBUG_RECD( status );
7348 
7349  // OK, received something; decrement incoming counter
7350  incoming1--;
7351  bool done = false;
7352 
7353  // In case ind is for ack, we need index of one before it
7354  unsigned int base_ind = 3 * ( ind / 3 );
7355  result = recv_buffer( MB_MESG_ENTS_SIZE, status, remoteOwnedBuffs[ind / 3], recv_ent_reqs[base_ind + 1],
7356  recv_ent_reqs[base_ind + 2], incoming1, localOwnedBuffs[ind / 3], sendReqs[base_ind + 1],
7357  sendReqs[base_ind + 2], done, ( store_remote_handles ? localOwnedBuffs[ind / 3] : NULL ),
7358  MB_MESG_REMOTEH_SIZE, &recv_remoteh_reqs[base_ind + 1], &incoming2 );
7359  MB_CHK_SET_ERR( result, "Failed to receive buffer" );
7360 
7361  if( done )
7362  {
7363  if( myDebug->get_verbosity() == 4 )
7364  {
7365  msgs.resize( msgs.size() + 1 );
7366  msgs.back() = new Buffer( *remoteOwnedBuffs[ind / 3] );
7367  }
7368 
7369  // Message completely received - process buffer that was sent
7370  remoteOwnedBuffs[ind / 3]->reset_ptr( sizeof( int ) );
7371  result = unpack_buffer( remoteOwnedBuffs[ind / 3]->buff_ptr, store_remote_handles, buffProcs[ind / 3],
7372  ind / 3, L1hloc, L1hrem, L1p, L2hloc, L2hrem, L2p, new_ents, true );
7373  if( MB_SUCCESS != result )
7374  {
7375  std::cout << "Failed to unpack entities. Buffer contents:" << std::endl;
7376  print_buffer( remoteOwnedBuffs[ind / 3]->mem_ptr, MB_MESG_ENTS_SIZE, buffProcs[ind / 3], false );
7377  return result;
7378  }
7379 
7380  if( recv_ent_reqs.size() != 3 * buffProcs.size() )
7381  {
7382  // Post irecv's for remote handles from new proc
7383  recv_remoteh_reqs.resize( 3 * buffProcs.size(), MPI_REQUEST_NULL );
7384  for( i = recv_ent_reqs.size(); i < 3 * buffProcs.size(); i += 3 )
7385  {
7386  localOwnedBuffs[i / 3]->reset_buffer();
7387  incoming2++;
7388  PRINT_DEBUG_IRECV( procConfig.proc_rank(), buffProcs[i / 3], localOwnedBuffs[i / 3]->mem_ptr,
7389  INITIAL_BUFF_SIZE, MB_MESG_REMOTEH_SIZE, incoming2 );
7390  success = MPI_Irecv( localOwnedBuffs[i / 3]->mem_ptr, INITIAL_BUFF_SIZE, MPI_UNSIGNED_CHAR,
7391  buffProcs[i / 3], MB_MESG_REMOTEH_SIZE, procConfig.proc_comm(),
7392  &recv_remoteh_reqs[i] );
7393  if( success != MPI_SUCCESS )
7394  {
7395  MB_SET_ERR( MB_FAILURE, "Failed to post irecv for remote handles in ghost exchange" );
7396  }
7397  }
7398  recv_ent_reqs.resize( 3 * buffProcs.size(), MPI_REQUEST_NULL );
7399  sendReqs.resize( 3 * buffProcs.size(), MPI_REQUEST_NULL );
7400  }
7401  }
7402  }
7403 
7404  // Assign and remove newly created elements from/to receive processor
7405  result = assign_entities_part( new_ents, procConfig.proc_rank() );
7406  MB_CHK_SET_ERR( result, "Failed to assign entities to part" );
7407  if( migrate )
7408  {
7409  result = remove_entities_part( allsent, procConfig.proc_rank() );
7410  MB_CHK_SET_ERR( result, "Failed to remove entities to part" );
7411  }
7412 
7413  // Add requests for any new addl procs
7414  if( recv_ent_reqs.size() != 3 * buffProcs.size() )
7415  {
7416  // Shouldn't get here...
7417  MB_SET_ERR( MB_FAILURE, "Requests length doesn't match proc count in entity exchange" );
7418  }
7419 
7420 #ifdef MOAB_HAVE_MPE
7421  if( myDebug->get_verbosity() == 2 )
7422  {
7423  MPE_Log_event( ENTITIES_END, procConfig.proc_rank(), "Ending entity exchange." );
7424  }
7425 #endif
7426 
7427  // we still need to wait on sendReqs, if they are not fulfilled yet
7428  if( wait_all )
7429  {
7430  if( myDebug->get_verbosity() == 5 )
7431  {
7432  success = MPI_Barrier( procConfig.proc_comm() );
7433  }
7434  else
7435  {
7436  MPI_Status mult_status[3 * MAX_SHARING_PROCS];
7437  success = MPI_Waitall( 3 * buffProcs.size(), sendReqs.data(), mult_status );
7438  if( MPI_SUCCESS != success )
7439  {
7440  MB_SET_ERR( MB_FAILURE, "Failed in waitall in exchange owned mesh" );
7441  }
7442  }
7443  }
7444 
7445  //===========================================
7446  // Send local handles for new entity to owner
7447  //===========================================
7448  for( i = 0; i < n_proc; i++ )
7449  {
7450  ind = get_buffers( exchange_procs[i] );
7451  // Reserve space on front for size and for initial buff size
7452  remoteOwnedBuffs[ind]->reset_buffer( sizeof( int ) );
7453 
7454  result = pack_remote_handles( L1hloc[ind], L1hrem[ind], L1p[ind], buffProcs[ind], remoteOwnedBuffs[ind] );
7455  MB_CHK_SET_ERR( result, "Failed to pack remote handles" );
7456  remoteOwnedBuffs[ind]->set_stored_size();
7457 
7458  if( myDebug->get_verbosity() == 4 )
7459  {
7460  msgs.resize( msgs.size() + 1 );
7461  msgs.back() = new Buffer( *remoteOwnedBuffs[ind] );
7462  }
7463  result = send_buffer( buffProcs[ind], remoteOwnedBuffs[ind], MB_MESG_REMOTEH_SIZE, sendReqs[3 * ind],
7464  recv_remoteh_reqs[3 * ind + 2], &dum_ack_buff, incoming2 );
7465  MB_CHK_SET_ERR( result, "Failed to send remote handles" );
7466  }
7467 
7468  //===========================================
7469  // Process remote handles of my ghosteds
7470  //===========================================
7471  while( incoming2 )
7472  {
7473  PRINT_DEBUG_WAITANY( recv_remoteh_reqs, MB_MESG_REMOTEH_SIZE, procConfig.proc_rank() );
7474  success = MPI_Waitany( 3 * buffProcs.size(), recv_remoteh_reqs.data(), &ind, &status );
7475  if( MPI_SUCCESS != success )
7476  {
7477  MB_SET_ERR( MB_FAILURE, "Failed in waitany in owned entity exchange" );
7478  }
7479 
7480  // OK, received something; decrement incoming counter
7481  incoming2--;
7482 
7483  PRINT_DEBUG_RECD( status );
7484 
7485  bool done = false;
7486  unsigned int base_ind = 3 * ( ind / 3 );
7487  result = recv_buffer( MB_MESG_REMOTEH_SIZE, status, localOwnedBuffs[ind / 3], recv_remoteh_reqs[base_ind + 1],
7488  recv_remoteh_reqs[base_ind + 2], incoming2, remoteOwnedBuffs[ind / 3],
7489  sendReqs[base_ind + 1], sendReqs[base_ind + 2], done );
7490  MB_CHK_SET_ERR( result, "Failed to receive remote handles" );
7491 
7492  if( done )
7493  {
7494  // Incoming remote handles
7495  if( myDebug->get_verbosity() == 4 )
7496  {
7497  msgs.resize( msgs.size() + 1 );
7498  msgs.back() = new Buffer( *localOwnedBuffs[ind / 3] );
7499  }
7500 
7501  localOwnedBuffs[ind / 3]->reset_ptr( sizeof( int ) );
7502  result =
7503  unpack_remote_handles( buffProcs[ind / 3], localOwnedBuffs[ind / 3]->buff_ptr, L2hloc, L2hrem, L2p );
7504  MB_CHK_SET_ERR( result, "Failed to unpack remote handles" );
7505  }
7506  }
7507 
7508 #ifdef MOAB_HAVE_MPE
7509  if( myDebug->get_verbosity() == 2 )
7510  {
7511  MPE_Log_event( RHANDLES_END, procConfig.proc_rank(), "Ending remote handles." );
7512  MPE_Log_event( OWNED_END, procConfig.proc_rank(), "Ending ghost exchange (still doing checks)." );
7513  }
7514 #endif
7515 
7516  //===========================================
7517  // Wait if requested
7518  //===========================================
7519  if( wait_all )
7520  {
7521  if( myDebug->get_verbosity() == 5 )
7522  {
7523  success = MPI_Barrier( procConfig.proc_comm() );
7524  }
7525  else
7526  {
7527  MPI_Status mult_status[3 * MAX_SHARING_PROCS];
7528  success = MPI_Waitall( 3 * buffProcs.size(), recv_remoteh_reqs.data(), mult_status );
7529  if( MPI_SUCCESS == success ) success = MPI_Waitall( 3 * buffProcs.size(), sendReqs.data(), mult_status );
7530  }
7531  if( MPI_SUCCESS != success )
7532  {
7533  MB_SET_ERR( MB_FAILURE, "Failed in waitall in owned entity exchange" );
7534  }
7535  }
7536 
7537 #ifndef NDEBUG
7538  result = check_sent_ents( allsent );
7539  MB_CHK_SET_ERR( result, "Failed check on shared entities" );
7540 #endif
7541  myDebug->tprintf( 1, "Exiting exchange_owned_mesh\n" );
7542 
7543  return MB_SUCCESS;
7544 }

References add_verts(), assign_entities_part(), buffProcs, check_sent_ents(), moab::Range::compactness(), moab::Range::empty(), moab::TupleList::enableWriteAccess(), ErrorCode, filter_pstatus(), get_buffers(), moab::TupleList::get_n(), moab::DebugOutput::get_verbosity(), moab::TupleList::inc_n(), INITIAL_BUFF_SIZE, moab::TupleList::initialize(), localOwnedBuffs, MAX_SHARING_PROCS, MB_CHK_SET_ERR, moab::MB_MESG_ENTS_SIZE, moab::MB_MESG_REMOTEH_SIZE, MB_SET_ERR, MB_SUCCESS, moab::Range::merge(), MPE_Log_event, moab::msgs, myDebug, pack_buffer(), pack_remote_handles(), print_buffer(), PRINT_DEBUG_IRECV, PRINT_DEBUG_RECD, PRINT_DEBUG_WAITANY, moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_rank(), procConfig, PSTATUS_AND, PSTATUS_SHARED, recv_buffer(), remoteOwnedBuffs, remove_entities_part(), moab::TupleList::buffer::reset(), moab::TupleList::reset(), reset_all_buffers(), send_buffer(), sendReqs, moab::Range::size(), size(), moab::TupleList::sort(), moab::subtract(), moab::DebugOutput::tprintf(), unpack_buffer(), unpack_remote_handles(), moab::TupleList::vi_wr, and moab::TupleList::vul_wr.

Referenced by exchange_owned_meshs().

◆ exchange_owned_meshs()

ErrorCode moab::ParallelComm::exchange_owned_meshs ( std::vector< unsigned int > &  exchange_procs,
std::vector< Range * > &  exchange_ents,
std::vector< MPI_Request > &  recv_ent_reqs,
std::vector< MPI_Request > &  recv_remoteh_reqs,
bool  store_remote_handles,
bool  wait_all = true,
bool  migrate = false,
int  dim = 0 
)

Exchange owned mesh for input mesh entities and sets This function should be called collectively over the communicator for this ParallelComm. If this version is called, all shared exchanged entities should have a value for this tag (or the tag should have a default value).

Parameters
exchange_procsprocessor vector exchanged
exchange_entsexchanged entities for each processors
migrateif the owner if entities are changed or not

Definition at line 7097 of file ParallelComm.cpp.

7105 {
7106  // Filter out entities already shared with destination
7107  // Exchange twice for entities and sets
7108  ErrorCode result;
7109  std::vector< unsigned int > exchange_procs_sets;
7110  std::vector< Range* > exchange_sets;
7111  int n_proc = exchange_procs.size();
7112  for( int i = 0; i < n_proc; i++ )
7113  {
7114  Range set_range = exchange_ents[i]->subset_by_type( MBENTITYSET );
7115  *exchange_ents[i] = subtract( *exchange_ents[i], set_range );
7116  Range* tmp_range = new Range( set_range );
7117  exchange_sets.push_back( tmp_range );
7118  exchange_procs_sets.push_back( exchange_procs[i] );
7119  }
7120 
7121  if( dim == 2 )
7122  {
7123  // Exchange entities first
7124  result = exchange_owned_mesh( exchange_procs, exchange_ents, recvReqs, recvRemotehReqs, true,
7125  store_remote_handles, wait_all, migrate );
7126  MB_CHK_SET_ERR( result, "Failed to exchange owned mesh entities" );
7127 
7128  // Exchange sets
7129  result = exchange_owned_mesh( exchange_procs_sets, exchange_sets, recvReqs, recvRemotehReqs, false,
7130  store_remote_handles, wait_all, migrate );
7131  }
7132  else
7133  {
7134  // Exchange entities first
7135  result = exchange_owned_mesh( exchange_procs, exchange_ents, recv_ent_reqs, recv_remoteh_reqs, false,
7136  store_remote_handles, wait_all, migrate );
7137  MB_CHK_SET_ERR( result, "Failed to exchange owned mesh entities" );
7138 
7139  // Exchange sets
7140  result = exchange_owned_mesh( exchange_procs_sets, exchange_sets, recv_ent_reqs, recv_remoteh_reqs, false,
7141  store_remote_handles, wait_all, migrate );
7142  MB_CHK_SET_ERR( result, "Failed to exchange owned mesh sets" );
7143  }
7144 
7145  for( int i = 0; i < n_proc; i++ )
7146  delete exchange_sets[i];
7147 
7148  // Build up the list of shared entities
7149  std::map< std::vector< int >, std::vector< EntityHandle > > proc_nvecs;
7150  int procs[MAX_SHARING_PROCS];
7151  EntityHandle handles[MAX_SHARING_PROCS];
7152  int nprocs;
7153  unsigned char pstat;
7154  for( std::set< EntityHandle >::iterator vit = sharedEnts.begin(); vit != sharedEnts.end(); ++vit )
7155  {
7156  if( mbImpl->dimension_from_handle( *vit ) > 2 ) continue;
7157  result = get_sharing_data( *vit, procs, handles, pstat, nprocs );
7158  MB_CHK_SET_ERR( result, "Failed to get sharing data in exchange_owned_meshs" );
7159  std::sort( procs, procs + nprocs );
7160  std::vector< int > tmp_procs( procs, procs + nprocs );
7161  assert( tmp_procs.size() != 2 );
7162  proc_nvecs[tmp_procs].push_back( *vit );
7163  }
7164 
7165  // Create interface sets from shared entities
7166  result = create_interface_sets( proc_nvecs );
7167  MB_CHK_SET_ERR( result, "Failed to create interface sets" );
7168 
7169  return MB_SUCCESS;
7170 }

References create_interface_sets(), moab::Interface::dimension_from_handle(), ErrorCode, exchange_owned_mesh(), get_sharing_data(), MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SUCCESS, MBENTITYSET, mbImpl, recvRemotehReqs, recvReqs, sharedEnts, moab::Range::subset_by_type(), and moab::subtract().

◆ exchange_tags() [1/3]

ErrorCode moab::ParallelComm::exchange_tags ( const char *  tag_name,
const Range &  entities 
)
inline

Exchange tags for all shared and ghosted entities This function should be called collectively over the communicator for this ParallelComm. If the entities vector is empty, all shared entities participate in the exchange. If a proc has no owned entities this function must still be called since it is collective.

Parameters
tag_nameName of tag to be exchanged
entitiesEntities for which tags are exchanged

Definition at line 1589 of file ParallelComm.hpp.

1590 {
1591  // get the tag handle
1592  std::vector< Tag > tags( 1 );
1593  ErrorCode result = mbImpl->tag_get_handle( tag_name, 0, MB_TYPE_OPAQUE, tags[0], MB_TAG_ANY );
1594  if( MB_SUCCESS != result )
1595  return result;
1596  else if( !tags[0] )
1597  return MB_TAG_NOT_FOUND;
1598 
1599  return exchange_tags( tags, tags, entities );
1600 }

References ErrorCode, exchange_tags(), MB_SUCCESS, MB_TAG_ANY, MB_TAG_NOT_FOUND, MB_TYPE_OPAQUE, mbImpl, and moab::Interface::tag_get_handle().

◆ exchange_tags() [2/3]

ErrorCode moab::ParallelComm::exchange_tags ( const std::vector< Tag > &  src_tags,
const std::vector< Tag > &  dst_tags,
const Range &  entities 
)

Exchange tags for all shared and ghosted entities This function should be called collectively over the communicator for this ParallelComm. If this version is called, all ghosted/shared entities should have a value for this tag (or the tag should have a default value). If the entities vector is empty, all shared entities participate in the exchange. If a proc has no owned entities this function must still be called since it is collective.

Parameters
src_tagsVector of tag handles to be exchanged
dst_tagsTag handles to store the tags on the non-owning procs
entitiesEntities for which tags are exchanged

Definition at line 7821 of file ParallelComm.cpp.

7824 {
7825  ErrorCode result;
7826  int success;
7827 
7828  myDebug->tprintf( 1, "Entering exchange_tags\n" );
7829 
7830  // Get all procs interfacing to this proc
7831  std::set< unsigned int > exch_procs;
7832  result = get_comm_procs( exch_procs );
7833 
7834  // Post ghost irecv's for all interface procs
7835  // Index requests the same as buffer/sharing procs indices
7836  std::vector< MPI_Request > recv_tag_reqs( 3 * buffProcs.size(), MPI_REQUEST_NULL );
7837  // sent_ack_reqs(buffProcs.size(), MPI_REQUEST_NULL);
7838  std::vector< unsigned int >::iterator sit;
7839  int ind;
7840 
7841  reset_all_buffers();
7842  int incoming = 0;
7843 
7844  for( ind = 0, sit = buffProcs.begin(); sit != buffProcs.end(); ++sit, ind++ )
7845  {
7846  incoming++;
7847  PRINT_DEBUG_IRECV( *sit, procConfig.proc_rank(), remoteOwnedBuffs[ind]->mem_ptr, INITIAL_BUFF_SIZE,
7848  MB_MESG_TAGS_SIZE, incoming );
7849 
7850  success = MPI_Irecv( remoteOwnedBuffs[ind]->mem_ptr, INITIAL_BUFF_SIZE, MPI_UNSIGNED_CHAR, *sit,
7851  MB_MESG_TAGS_SIZE, procConfig.proc_comm(), &recv_tag_reqs[3 * ind] );
7852  if( success != MPI_SUCCESS )
7853  {
7854  MB_SET_ERR( MB_FAILURE, "Failed to post irecv in ghost exchange" );
7855  }
7856  }
7857 
7858  // Pack and send tags from this proc to others
7859  // Make sendReqs vector to simplify initialization
7860  sendReqs.resize( 3 * buffProcs.size(), MPI_REQUEST_NULL );
7861 
7862  // Take all shared entities if incoming list is empty
7863  Range entities;
7864  if( entities_in.empty() )
7865  std::copy( sharedEnts.begin(), sharedEnts.end(), range_inserter( entities ) );
7866  else
7867  entities = entities_in;
7868 
7869  int dum_ack_buff;
7870 
7871  for( ind = 0, sit = buffProcs.begin(); sit != buffProcs.end(); ++sit, ind++ )
7872  {
7873  Range tag_ents = entities;
7874 
7875  // Get ents shared by proc *sit
7876  result = filter_pstatus( tag_ents, PSTATUS_SHARED, PSTATUS_AND, *sit );
7877  MB_CHK_SET_ERR( result, "Failed pstatus AND check" );
7878 
7879  // Remote nonowned entities
7880  if( !tag_ents.empty() )
7881  {
7882  result = filter_pstatus( tag_ents, PSTATUS_NOT_OWNED, PSTATUS_NOT );
7883  MB_CHK_SET_ERR( result, "Failed pstatus NOT check" );
7884  }
7885 
7886  // Pack-send; this also posts receives if store_remote_handles is true
7887  std::vector< Range > tag_ranges;
7888  for( std::vector< Tag >::const_iterator vit = src_tags.begin(); vit != src_tags.end(); ++vit )
7889  {
7890  const void* ptr;
7891  int sz;
7892  if( mbImpl->tag_get_default_value( *vit, ptr, sz ) != MB_SUCCESS )
7893  {
7894  Range tagged_ents;
7895  mbImpl->get_entities_by_type_and_tag( 0, MBMAXTYPE, &*vit, 0, 1, tagged_ents );
7896  tag_ranges.push_back( intersect( tag_ents, tagged_ents ) );
7897  }
7898  else
7899  {
7900  tag_ranges.push_back( tag_ents );
7901  }
7902  }
7903 
7904  // Pack the data
7905  // Reserve space on front for size and for initial buff size
7906  localOwnedBuffs[ind]->reset_ptr( sizeof( int ) );
7907 
7908  result = pack_tags( tag_ents, src_tags, dst_tags, tag_ranges, localOwnedBuffs[ind], true, *sit );
7909  MB_CHK_SET_ERR( result, "Failed to count buffer in pack_send_tag" );
7910 
7911  // Now send it
7912  result = send_buffer( *sit, localOwnedBuffs[ind], MB_MESG_TAGS_SIZE, sendReqs[3 * ind],
7913  recv_tag_reqs[3 * ind + 2], &dum_ack_buff, incoming );
7914  MB_CHK_SET_ERR( result, "Failed to send buffer" );
7915  }
7916 
7917  // Receive/unpack tags
7918  while( incoming )
7919  {
7920  MPI_Status status;
7921  int index_in_recv_requests;
7922  PRINT_DEBUG_WAITANY( recv_tag_reqs, MB_MESG_TAGS_SIZE, procConfig.proc_rank() );
7923  success = MPI_Waitany( 3 * buffProcs.size(), recv_tag_reqs.data(), &index_in_recv_requests, &status );
7924  if( MPI_SUCCESS != success )
7925  {
7926  MB_SET_ERR( MB_FAILURE, "Failed in waitany in tag exchange" );
7927  }
7928  // Processor index in the list is divided by 3
7929  ind = index_in_recv_requests / 3;
7930 
7931  PRINT_DEBUG_RECD( status );
7932 
7933  // OK, received something; decrement incoming counter
7934  incoming--;
7935 
7936  bool done = false;
7937  std::vector< EntityHandle > dum_vec;
7938  result = recv_buffer( MB_MESG_TAGS_SIZE, status, remoteOwnedBuffs[ind],
7939  recv_tag_reqs[3 * ind + 1], // This is for receiving the second message
7940  recv_tag_reqs[3 * ind + 2], // This would be for ack, but it is not
7941  // used; consider removing it
7942  incoming, localOwnedBuffs[ind],
7943  sendReqs[3 * ind + 1], // Send request for sending the second message
7944  sendReqs[3 * ind + 2], // This is for sending the ack
7945  done );
7946  MB_CHK_SET_ERR( result, "Failed to resize recv buffer" );
7947  if( done )
7948  {
7949  remoteOwnedBuffs[ind]->reset_ptr( sizeof( int ) );
7950  result = unpack_tags( remoteOwnedBuffs[ind]->buff_ptr, dum_vec, true, buffProcs[ind] );
7951  MB_CHK_SET_ERR( result, "Failed to recv-unpack-tag message" );
7952  }
7953  }
7954 
7955  // OK, now wait
7956  if( myDebug->get_verbosity() == 5 )
7957  {
7958  success = MPI_Barrier( procConfig.proc_comm() );
7959  }
7960  else
7961  {
7962  MPI_Status status[3 * MAX_SHARING_PROCS];
7963  success = MPI_Waitall( 3 * buffProcs.size(), sendReqs.data(), status );
7964  }
7965  if( MPI_SUCCESS != success )
7966  {
7967  MB_SET_ERR( MB_FAILURE, "Failure in waitall in tag exchange" );
7968  }
7969 
7970  // If source tag is not equal to destination tag, then
7971  // do local copy for owned entities (communicate w/ self)
7972  assert( src_tags.size() == dst_tags.size() );
7973  if( src_tags != dst_tags )
7974  {
7975  std::vector< unsigned char > data;
7976  Range owned_ents;
7977  if( entities_in.empty() )
7978  std::copy( sharedEnts.begin(), sharedEnts.end(), range_inserter( entities ) );
7979  else
7980  owned_ents = entities_in;
7981  result = filter_pstatus( owned_ents, PSTATUS_NOT_OWNED, PSTATUS_NOT );
7982  MB_CHK_SET_ERR( result, "Failure to get subset of owned entities" );
7983 
7984  if( !owned_ents.empty() )
7985  { // Check this here, otherwise we get
7986  // Unexpected results from get_entities_by_type_and_tag w/ Interface::INTERSECT
7987  for( size_t i = 0; i < src_tags.size(); i++ )
7988  {
7989  if( src_tags[i] == dst_tags[i] ) continue;
7990 
7991  Range tagged_ents( owned_ents );
7992  result = mbImpl->get_entities_by_type_and_tag( 0, MBMAXTYPE, src_tags.data(), 0, 1, tagged_ents,
7993  Interface::INTERSECT );
7994  MB_CHK_SET_ERR( result, "get_entities_by_type_and_tag(type == MBMAXTYPE) failed" );
7995 
7996  int sz, size2;
7997  result = mbImpl->tag_get_bytes( src_tags[i], sz );
7998  MB_CHK_SET_ERR( result, "tag_get_size failed" );
7999  result = mbImpl->tag_get_bytes( dst_tags[i], size2 );
8000  MB_CHK_SET_ERR( result, "tag_get_size failed" );
8001  if( sz != size2 )
8002  {
8003  MB_SET_ERR( MB_FAILURE, "tag sizes don't match" );
8004  }
8005 
8006  data.resize( sz * tagged_ents.size() );
8007  result = mbImpl->tag_get_data( src_tags[i], tagged_ents, data.data() );
8008  MB_CHK_SET_ERR( result, "tag_get_data failed" );
8009  result = mbImpl->tag_set_data( dst_tags[i], tagged_ents, data.data() );
8010  MB_CHK_SET_ERR( result, "tag_set_data failed" );
8011  }
8012  }
8013  }
8014 
8015  myDebug->tprintf( 1, "Exiting exchange_tags" );
8016 
8017  return MB_SUCCESS;
8018 }

References buffProcs, moab::Range::empty(), ErrorCode, filter_pstatus(), get_comm_procs(), moab::Interface::get_entities_by_type_and_tag(), moab::DebugOutput::get_verbosity(), INITIAL_BUFF_SIZE, moab::intersect(), moab::Interface::INTERSECT, localOwnedBuffs, MAX_SHARING_PROCS, MB_CHK_SET_ERR, moab::MB_MESG_TAGS_SIZE, MB_SET_ERR, MB_SUCCESS, mbImpl, MBMAXTYPE, myDebug, pack_tags(), PRINT_DEBUG_IRECV, PRINT_DEBUG_RECD, PRINT_DEBUG_WAITANY, moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_rank(), procConfig, PSTATUS_AND, PSTATUS_NOT, PSTATUS_NOT_OWNED, PSTATUS_SHARED, recv_buffer(), remoteOwnedBuffs, reset_all_buffers(), send_buffer(), sendReqs, sharedEnts, moab::Range::size(), moab::Interface::tag_get_bytes(), moab::Interface::tag_get_data(), moab::Interface::tag_get_default_value(), moab::Interface::tag_set_data(), moab::DebugOutput::tprintf(), and unpack_tags().

Referenced by assign_global_ids(), moab::WriteHDF5Parallel::exchange_file_ids(), moab::NestedRefine::exchange_ghosts(), exchange_tags(), iMOAB_SynchronizeTags(), main(), perform_laplacian_smoothing(), perform_lloyd_relaxation(), and moab::LloydSmoother::perform_smooth().

◆ exchange_tags() [3/3]

ErrorCode moab::ParallelComm::exchange_tags ( Tag  tagh,
const Range &  entities 
)
inline

Exchange tags for all shared and ghosted entities This function should be called collectively over the communicator for this ParallelComm. If the entities vector is empty, all shared entities participate in the exchange. If a proc has no owned entities this function must still be called since it is collective.

Parameters
taghHandle of tag to be exchanged
entitiesEntities for which tags are exchanged

Definition at line 1602 of file ParallelComm.hpp.

1603 {
1604  // get the tag handle
1605  std::vector< Tag > tags;
1606  tags.push_back( tagh );
1607 
1608  return exchange_tags( tags, tags, entities );
1609 }

References exchange_tags().

◆ filter_pstatus()

ErrorCode moab::ParallelComm::filter_pstatus ( Range &  ents,
const unsigned char  pstatus_val,
const unsigned char  op,
int  to_proc = -1,
Range *  returned_ents = NULL 
)

Filter the entities by pstatus tag. op is one of PSTATUS_ AND, OR, NOT; an entity is output if: AND: all bits set in pstatus_val are also set on entity OR: any bits set in pstatus_val also set on entity NOT: any bits set in pstatus_val are not set on entity

Results returned in input list, unless result_ents is passed in non-null, in which case results are returned in result_ents.

If ents is passed in empty, filter is done on shared entities in this pcomm instance, i.e. contents of sharedEnts.

Parameters
entsInput entities to filter
pstatus_valpstatus value to which entities are compared
opBitwise operation performed between pstatus values
to_procIf non-negative and PSTATUS_SHARED is set on pstatus_val, only entities shared with to_proc are returned
result_entsIf non-null, results of filter are put in the pointed-to range

Definition at line 5778 of file ParallelComm.cpp.

5783 {
5784  Range tmp_ents;
5785 
5786  // assert(!ents.empty());
5787  if( ents.empty() )
5788  {
5789  if( returned_ents ) returned_ents->clear();
5790  return MB_SUCCESS;
5791  }
5792 
5793  // Put into tmp_ents any entities which are not owned locally or
5794  // who are already shared with to_proc
5795  std::vector< unsigned char > shared_flags( ents.size() ), shared_flags2;
5796  ErrorCode result = mbImpl->tag_get_data( pstatus_tag(), ents, shared_flags.data() );
5797  MB_CHK_SET_ERR( result, "Failed to get pstatus flag" );
5798  Range::const_iterator rit, hint = tmp_ents.begin();
5799  ;
5800  int i;
5801  if( op == PSTATUS_OR )
5802  {
5803  for( rit = ents.begin(), i = 0; rit != ents.end(); ++rit, i++ )
5804  {
5805  if( ( ( shared_flags[i] & ~pstat ) ^ shared_flags[i] ) & pstat )
5806  {
5807  hint = tmp_ents.insert( hint, *rit );
5808  if( -1 != to_proc ) shared_flags2.push_back( shared_flags[i] );
5809  }
5810  }
5811  }
5812  else if( op == PSTATUS_AND )
5813  {
5814  for( rit = ents.begin(), i = 0; rit != ents.end(); ++rit, i++ )
5815  {
5816  if( ( shared_flags[i] & pstat ) == pstat )
5817  {
5818  hint = tmp_ents.insert( hint, *rit );
5819  if( -1 != to_proc ) shared_flags2.push_back( shared_flags[i] );
5820  }
5821  }
5822  }
5823  else if( op == PSTATUS_NOT )
5824  {
5825  for( rit = ents.begin(), i = 0; rit != ents.end(); ++rit, i++ )
5826  {
5827  if( !( shared_flags[i] & pstat ) )
5828  {
5829  hint = tmp_ents.insert( hint, *rit );
5830  if( -1 != to_proc ) shared_flags2.push_back( shared_flags[i] );
5831  }
5832  }
5833  }
5834  else
5835  {
5836  assert( false );
5837  return MB_FAILURE;
5838  }
5839 
5840  if( -1 != to_proc )
5841  {
5842  int sharing_procs[MAX_SHARING_PROCS];
5843  std::fill( sharing_procs, sharing_procs + MAX_SHARING_PROCS, -1 );
5844  Range tmp_ents2;
5845  hint = tmp_ents2.begin();
5846 
5847  for( rit = tmp_ents.begin(), i = 0; rit != tmp_ents.end(); ++rit, i++ )
5848  {
5849  // We need to check sharing procs
5850  if( shared_flags2[i] & PSTATUS_MULTISHARED )
5851  {
5852  result = mbImpl->tag_get_data( sharedps_tag(), &( *rit ), 1, sharing_procs );
5853  MB_CHK_SET_ERR( result, "Failed to get sharedps tag" );
5854  assert( -1 != sharing_procs[0] );
5855  for( unsigned int j = 0; j < MAX_SHARING_PROCS; j++ )
5856  {
5857  // If to_proc shares this entity, add it to list
5858  if( sharing_procs[j] == to_proc )
5859  {
5860  hint = tmp_ents2.insert( hint, *rit );
5861  }
5862  else if( -1 == sharing_procs[j] )
5863  break;
5864 
5865  sharing_procs[j] = -1;
5866  }
5867  }
5868  else if( shared_flags2[i] & PSTATUS_SHARED )
5869  {
5870  result = mbImpl->tag_get_data( sharedp_tag(), &( *rit ), 1, sharing_procs );
5871  MB_CHK_SET_ERR( result, "Failed to get sharedp tag" );
5872  assert( -1 != sharing_procs[0] );
5873  if( sharing_procs[0] == to_proc ) hint = tmp_ents2.insert( hint, *rit );
5874  sharing_procs[0] = -1;
5875  }
5876  else
5877  assert( "should never get here" && false );
5878  }
5879 
5880  tmp_ents.swap( tmp_ents2 );
5881  }
5882 
5883  if( returned_ents )
5884  returned_ents->swap( tmp_ents );
5885  else
5886  ents.swap( tmp_ents );
5887 
5888  return MB_SUCCESS;
5889 }

References moab::Range::begin(), moab::Range::clear(), moab::Range::empty(), moab::Range::end(), ErrorCode, moab::Range::insert(), MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, PSTATUS_AND, PSTATUS_MULTISHARED, PSTATUS_NOT, PSTATUS_OR, PSTATUS_SHARED, pstatus_tag(), sharedp_tag(), sharedps_tag(), moab::Range::size(), moab::Range::swap(), and moab::Interface::tag_get_data().

Referenced by moab::NCWriteGCRM::collect_mesh_info(), moab::ScdNCWriteHelper::collect_mesh_info(), moab::NCWriteHOMME::collect_mesh_info(), moab::NCWriteMPAS::collect_mesh_info(), moab::ScdNCHelper::create_quad_coordinate_tag(), moab::WriteHDF5Parallel::exchange_file_ids(), exchange_owned_mesh(), exchange_tags(), moab::WriteHDF5Parallel::gather_interface_meshes(), get_ghosted_entities(), get_max_volume(), get_sent_ents(), get_shared_entities(), hcFilter(), iMOAB_UpdateMeshInfo(), moab::HalfFacetRep::initialize(), moab::LloydSmoother::initialize(), moab::HiReconstruction::initialize(), laplacianFilter(), moab::ReadParallel::load_file(), main(), perform_laplacian_smoothing(), perform_lloyd_relaxation(), moab::LloydSmoother::perform_smooth(), moab::ScdNCHelper::read_scd_variables_to_nonset_allocate(), moab::NCHelperGCRM::read_ucd_variables_to_nonset_allocate(), moab::NCHelperMPAS::read_ucd_variables_to_nonset_allocate(), reduce_tags(), resolve_shared_sets(), send_entities(), and settle_intersection_points().

◆ find_existing_entity()

ErrorCode moab::ParallelComm::find_existing_entity ( const bool  is_iface,
const int  owner_p,
const EntityHandle  owner_h,
const int  num_ents,
const EntityHandle *  connect,
const int  num_connect,
const EntityType  this_type,
std::vector< EntityHandle > &  L2hloc,
std::vector< EntityHandle > &  L2hrem,
std::vector< unsigned int > &  L2p,
EntityHandle &  new_h 
)
private

given connectivity and type, find an existing entity, if there is one

Definition at line 3134 of file ParallelComm.cpp.

3145 {
3146  new_h = 0;
3147  if( !is_iface && num_ps > 2 )
3148  {
3149  for( unsigned int i = 0; i < L2hrem.size(); i++ )
3150  {
3151  if( L2hrem[i] == owner_h && owner_p == (int)L2p[i] )
3152  {
3153  new_h = L2hloc[i];
3154  return MB_SUCCESS;
3155  }
3156  }
3157  }
3158 
3159  // If we got here and it's a vertex, we don't need to look further
3160  if( MBVERTEX == this_type || !connect || !num_connect ) return MB_SUCCESS;
3161 
3162  Range tmp_range;
3163  ErrorCode result = mbImpl->get_adjacencies( connect, num_connect, CN::Dimension( this_type ), false, tmp_range );
3164  MB_CHK_SET_ERR( result, "Failed to get existing entity" );
3165  if( !tmp_range.empty() )
3166  {
3167  // Found a corresponding entity - return target
3168  new_h = *tmp_range.begin();
3169  }
3170  else
3171  {
3172  new_h = 0;
3173  }
3174 
3175  return MB_SUCCESS;
3176 }

References moab::Range::begin(), moab::CN::Dimension(), moab::Range::empty(), ErrorCode, moab::Interface::get_adjacencies(), MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, and MBVERTEX.

Referenced by unpack_entities(), and unpack_remote_handles().

◆ gather_data()

ErrorCode moab::ParallelComm::gather_data ( Range &  gather_ents,
Tag &  tag_handle,
Tag  id_tag = 0,
EntityHandle  gather_set = 0,
int  root_proc_rank = 0 
)

Definition at line 9248 of file ParallelComm.cpp.

9253 {
9254  int dim = mbImpl->dimension_from_handle( *gather_ents.begin() );
9255  int bytes_per_tag = 0;
9256  ErrorCode rval = mbImpl->tag_get_bytes( tag_handle, bytes_per_tag );
9257  if( rval != MB_SUCCESS ) return rval;
9258 
9259  int sz_buffer = sizeof( int ) + gather_ents.size() * ( sizeof( int ) + bytes_per_tag );
9260  void* senddata = malloc( sz_buffer );
9261  ( (int*)senddata )[0] = (int)gather_ents.size();
9262  int* ptr_int = (int*)senddata + 1;
9263  rval = mbImpl->tag_get_data( id_tag, gather_ents, (void*)ptr_int );
9264  if( rval != MB_SUCCESS ) return rval;
9265  ptr_int = (int*)( senddata ) + 1 + gather_ents.size();
9266  rval = mbImpl->tag_get_data( tag_handle, gather_ents, (void*)ptr_int );
9267  if( rval != MB_SUCCESS ) return rval;
9268  std::vector< int > displs( proc_config().proc_size(), 0 );
9269  MPI_Gather( &sz_buffer, 1, MPI_INT, displs.data(), 1, MPI_INT, root_proc_rank, comm() );
9270  std::vector< int > recvcnts( proc_config().proc_size(), 0 );
9271  std::copy( displs.begin(), displs.end(), recvcnts.begin() );
9272  std::partial_sum( displs.begin(), displs.end(), displs.begin() );
9273  std::vector< int >::iterator lastM1 = displs.end() - 1;
9274  std::copy_backward( displs.begin(), lastM1, displs.end() );
9275  // std::copy_backward(displs.begin(), --displs.end(), displs.end());
9276  displs[0] = 0;
9277 
9278  if( (int)rank() != root_proc_rank )
9279  MPI_Gatherv( senddata, sz_buffer, MPI_BYTE, NULL, NULL, NULL, MPI_BYTE, root_proc_rank, comm() );
9280  else
9281  {
9282  Range gents;
9283  mbImpl->get_entities_by_dimension( gather_set, dim, gents );
9284  int recvbuffsz = gents.size() * ( bytes_per_tag + sizeof( int ) ) + proc_config().proc_size() * sizeof( int );
9285  void* recvbuf = malloc( recvbuffsz );
9286  MPI_Gatherv( senddata, sz_buffer, MPI_BYTE, recvbuf, recvcnts.data(), displs.data(), MPI_BYTE, root_proc_rank,
9287  comm() );
9288 
9289  void* gvals = NULL;
9290 
9291  // Test whether gents has multiple sequences
9292  bool multiple_sequences = false;
9293  if( gents.psize() > 1 )
9294  multiple_sequences = true;
9295  else
9296  {
9297  int count;
9298  rval = mbImpl->tag_iterate( tag_handle, gents.begin(), gents.end(), count, gvals );
9299  assert( NULL != gvals );
9300  assert( count > 0 );
9301  if( (size_t)count != gents.size() )
9302  {
9303  multiple_sequences = true;
9304  gvals = NULL;
9305  }
9306  }
9307 
9308  // If gents has multiple sequences, create a temp buffer for gathered values
9309  if( multiple_sequences )
9310  {
9311  gvals = malloc( gents.size() * bytes_per_tag );
9312  assert( NULL != gvals );
9313  }
9314 
9315  for( int i = 0; i != (int)size(); i++ )
9316  {
9317  int numents = *(int*)( ( (char*)recvbuf ) + displs[i] );
9318  int* id_ptr = (int*)( ( (char*)recvbuf ) + displs[i] + sizeof( int ) );
9319  char* val_ptr = (char*)( id_ptr + numents );
9320  for( int j = 0; j != numents; j++ )
9321  {
9322  int idx = id_ptr[j];
9323  memcpy( (char*)gvals + ( idx - 1 ) * bytes_per_tag, val_ptr + j * bytes_per_tag, bytes_per_tag );
9324  }
9325  }
9326 
9327  // Free the receive buffer
9328  free( recvbuf );
9329 
9330  // If gents has multiple sequences, copy tag data (stored in the temp buffer) to each
9331  // sequence separately
9332  if( multiple_sequences )
9333  {
9334  Range::iterator iter = gents.begin();
9335  size_t start_idx = 0;
9336  while( iter != gents.end() )
9337  {
9338  int count;
9339  void* ptr;
9340  rval = mbImpl->tag_iterate( tag_handle, iter, gents.end(), count, ptr );
9341  assert( NULL != ptr );
9342  assert( count > 0 );
9343  memcpy( (char*)ptr, (char*)gvals + start_idx * bytes_per_tag, bytes_per_tag * count );
9344 
9345  iter += count;
9346  start_idx += count;
9347  }
9348  assert( start_idx == gents.size() );
9349 
9350  // Free the temp buffer
9351  free( gvals );
9352  }
9353  }
9354 
9355  // Free the send data
9356  free( senddata );
9357 
9358  return MB_SUCCESS;
9359 }

References moab::Range::begin(), comm(), moab::Interface::dimension_from_handle(), moab::Range::end(), ErrorCode, moab::Interface::get_entities_by_dimension(), MB_SUCCESS, mbImpl, proc_config(), moab::ProcConfig::proc_size(), moab::Range::psize(), rank(), moab::Range::size(), size(), moab::Interface::tag_get_bytes(), moab::Interface::tag_get_data(), and moab::Interface::tag_iterate().

◆ get_all_pcomm()

ErrorCode moab::ParallelComm::get_all_pcomm ( Interface *  impl,
std::vector< ParallelComm * > &  list 
)
static

Definition at line 8342 of file ParallelComm.cpp.

8343 {
8344  Tag pc_tag = pcomm_tag( impl, false );
8345  if( 0 == pc_tag ) return MB_TAG_NOT_FOUND;
8346 
8347  const EntityHandle root = 0;
8348  ParallelComm* pc_array[MAX_SHARING_PROCS];
8349  ErrorCode rval = impl->tag_get_data( pc_tag, &root, 1, pc_array );
8350  if( MB_SUCCESS != rval ) return rval;
8351 
8352  for( int i = 0; i < MAX_SHARING_PROCS; i++ )
8353  {
8354  if( pc_array[i] ) list.push_back( pc_array[i] );
8355  }
8356 
8357  return MB_SUCCESS;
8358 }

References ErrorCode, MAX_SHARING_PROCS, MB_SUCCESS, MB_TAG_NOT_FOUND, pcomm_tag(), and moab::Interface::tag_get_data().

Referenced by moab::Core::deinitialize().

◆ get_buffers()

int moab::ParallelComm::get_buffers ( int  to_proc,
bool *  is_new = NULL 
)

get (and possibly allocate) buffers for messages to/from to_proc; returns index of to_proc in buffProcs vector; if is_new is non-NULL, sets to whether new buffer was allocated PUBLIC ONLY FOR TESTING!

Definition at line 517 of file ParallelComm.cpp.

518 {
519  int ind = -1;
520  std::vector< unsigned int >::iterator vit = std::find( buffProcs.begin(), buffProcs.end(), to_proc );
521  if( vit == buffProcs.end() )
522  {
523  assert( "shouldn't need buffer to myself" && to_proc != (int)procConfig.proc_rank() );
524  ind = buffProcs.size();
525  buffProcs.push_back( (unsigned int)to_proc );
526  localOwnedBuffs.push_back( new Buffer( INITIAL_BUFF_SIZE ) );
527  remoteOwnedBuffs.push_back( new Buffer( INITIAL_BUFF_SIZE ) );
528  if( is_new ) *is_new = true;
529  }
530  else
531  {
532  ind = vit - buffProcs.begin();
533  if( is_new ) *is_new = false;
534  }
535  assert( ind < MAX_SHARING_PROCS );
536  return ind;
537 }

References buffProcs, INITIAL_BUFF_SIZE, localOwnedBuffs, MAX_SHARING_PROCS, moab::ProcConfig::proc_rank(), procConfig, and remoteOwnedBuffs.

Referenced by check_all_shared_handles(), correct_thin_ghost_layers(), exchange_owned_mesh(), get_interface_procs(), pack_shared_handles(), post_irecv(), recv_entities(), recv_messages(), recv_remote_handle_messages(), send_entities(), send_recv_entities(), moab::ScdInterface::tag_shared_vertices(), and unpack_entities().

◆ get_comm_procs()

ErrorCode moab::ParallelComm::get_comm_procs ( std::set< unsigned int > &  procs)
inline

get processors with which this processor communicates

Definition at line 1633 of file ParallelComm.hpp.

1634 {
1635  ErrorCode result = get_interface_procs( procs );
1636  if( MB_SUCCESS != result ) return result;
1637 
1638  std::copy( buffProcs.begin(), buffProcs.end(), std::inserter( procs, procs.begin() ) );
1639 
1640  return MB_SUCCESS;
1641 }

References buffProcs, ErrorCode, get_interface_procs(), and MB_SUCCESS.

Referenced by exchange_tags(), reduce_tags(), and settle_intersection_points().

◆ get_debug_verbosity()

int moab::ParallelComm::get_debug_verbosity ( )

get the verbosity level of output from this pcomm

Definition at line 9206 of file ParallelComm.cpp.

9207 {
9208  return myDebug->get_verbosity();
9209 }

References moab::DebugOutput::get_verbosity(), and myDebug.

Referenced by augment_default_sets_with_ghosts(), moab::ScdInterface::construct_box(), and moab::ScdInterface::tag_shared_vertices().

◆ get_entityset_local_handle()

ErrorCode moab::ParallelComm::get_entityset_local_handle ( unsigned  owning_rank,
EntityHandle  remote_handle,
EntityHandle &  local_handle 
) const

Given set owner and handle on owner, find local set handle.

Definition at line 9226 of file ParallelComm.cpp.

9229 {
9230  return sharedSetData->get_local_handle( owning_rank, remote_handle, local_handle );
9231 }

References moab::SharedSetData::get_local_handle(), and sharedSetData.

Referenced by moab::WriteHDF5Parallel::communicate_shared_set_ids().

◆ get_entityset_owner()

ErrorCode moab::ParallelComm::get_entityset_owner ( EntityHandle  entity_set,
unsigned &  owner_rank,
EntityHandle *  remote_handle = 0 
) const

Get rank of the owner of a shared set. Returns this proc if set is not shared. Optionally returns handle on owning process for shared set.

Definition at line 9216 of file ParallelComm.cpp.

9219 {
9220  if( remote_handle )
9221  return sharedSetData->get_owner( entity_set, owner_rank, *remote_handle );
9222  else
9223  return sharedSetData->get_owner( entity_set, owner_rank );
9224 }

References moab::SharedSetData::get_owner(), and sharedSetData.

Referenced by moab::WriteHDF5Parallel::communicate_shared_set_data(), moab::WriteHDF5Parallel::communicate_shared_set_ids(), and moab::WriteHDF5Parallel::print_set_sharing_data().

◆ get_entityset_owners()

ErrorCode moab::ParallelComm::get_entityset_owners ( std::vector< unsigned > &  ranks) const

Get ranks of all processes that own at least one set that is shared with this process. Will include the rank of this process if this process owns any shared set.

Definition at line 9238 of file ParallelComm.cpp.

9239 {
9240  return sharedSetData->get_owning_procs( ranks );
9241 }

References moab::SharedSetData::get_owning_procs(), and sharedSetData.

Referenced by moab::WriteHDF5Parallel::communicate_shared_set_ids().

◆ get_entityset_procs()

ErrorCode moab::ParallelComm::get_entityset_procs ( EntityHandle  entity_set,
std::vector< unsigned > &  ranks 
) const

Get array of process IDs sharing a set. Returns zero and passes back NULL if set is not shared.

Definition at line 9211 of file ParallelComm.cpp.

9212 {
9213  return sharedSetData->get_sharing_procs( set, ranks );
9214 }

References moab::SharedSetData::get_sharing_procs(), and sharedSetData.

Referenced by moab::WriteHDF5Parallel::communicate_shared_set_data(), moab::WriteHDF5Parallel::communicate_shared_set_ids(), and moab::WriteHDF5Parallel::print_set_sharing_data().

◆ get_ghosted_entities()

ErrorCode moab::ParallelComm::get_ghosted_entities ( int  bridge_dim,
int  ghost_dim,
int  to_proc,
int  num_layers,
int  addl_ents,
Range &  ghosted_ents 
)
private

for specified bridge/ghost dimension, to_proc, and number of layers, get the entities to be ghosted, and info on additional procs needing to communicate with to_proc

Definition at line 7720 of file ParallelComm.cpp.

7726 {
7727  // Get bridge ents on interface(s)
7728  Range from_ents;
7729  ErrorCode result = MB_SUCCESS;
7730  assert( 0 < num_layers );
7731  for( Range::iterator rit = interfaceSets.begin(); rit != interfaceSets.end(); ++rit )
7732  {
7733  if( !is_iface_proc( *rit, to_proc ) ) continue;
7734 
7735  // Get starting "from" entities
7736  if( bridge_dim == -1 )
7737  {
7738  result = mbImpl->get_entities_by_handle( *rit, from_ents );
7739  MB_CHK_SET_ERR( result, "Failed to get bridge ents in the set" );
7740  }
7741  else
7742  {
7743  result = mbImpl->get_entities_by_dimension( *rit, bridge_dim, from_ents );
7744  MB_CHK_SET_ERR( result, "Failed to get bridge ents in the set" );
7745  }
7746 
7747  // Need to get layers of bridge-adj entities
7748  if( from_ents.empty() ) continue;
7749  result =
7750  MeshTopoUtil( mbImpl ).get_bridge_adjacencies( from_ents, bridge_dim, ghost_dim, ghosted_ents, num_layers );
7751  MB_CHK_SET_ERR( result, "Failed to get bridge adjacencies" );
7752  }
7753 
7754  result = add_verts( ghosted_ents );
7755  MB_CHK_SET_ERR( result, "Failed to add verts" );
7756 
7757  if( addl_ents )
7758  {
7759  // First get the ents of ghost_dim
7760  Range tmp_ents, tmp_owned, tmp_notowned;
7761  tmp_owned = ghosted_ents.subset_by_dimension( ghost_dim );
7762  if( tmp_owned.empty() ) return result;
7763 
7764  tmp_notowned = tmp_owned;
7765 
7766  // Next, filter by pstatus; can only create adj entities for entities I own
7767  result = filter_pstatus( tmp_owned, PSTATUS_NOT_OWNED, PSTATUS_NOT, -1, &tmp_owned );
7768  MB_CHK_SET_ERR( result, "Failed to filter owned entities" );
7769 
7770  tmp_notowned -= tmp_owned;
7771 
7772  // Get edges first
7773  if( 1 == addl_ents || 3 == addl_ents )
7774  {
7775  result = mbImpl->get_adjacencies( tmp_owned, 1, true, tmp_ents, Interface::UNION );
7776  MB_CHK_SET_ERR( result, "Failed to get edge adjacencies for owned ghost entities" );
7777  result = mbImpl->get_adjacencies( tmp_notowned, 1, false, tmp_ents, Interface::UNION );
7778  MB_CHK_SET_ERR( result, "Failed to get edge adjacencies for notowned ghost entities" );
7779  }
7780  if( 2 == addl_ents || 3 == addl_ents )
7781  {
7782  result = mbImpl->get_adjacencies( tmp_owned, 2, true, tmp_ents, Interface::UNION );
7783  MB_CHK_SET_ERR( result, "Failed to get face adjacencies for owned ghost entities" );
7784  result = mbImpl->get_adjacencies( tmp_notowned, 2, false, tmp_ents, Interface::UNION );
7785  MB_CHK_SET_ERR( result, "Failed to get face adjacencies for notowned ghost entities" );
7786  }
7787 
7788  ghosted_ents.merge( tmp_ents );
7789  }
7790 
7791  return result;
7792 }

References add_verts(), moab::Range::begin(), moab::Range::empty(), moab::Range::end(), ErrorCode, filter_pstatus(), moab::Interface::get_adjacencies(), moab::MeshTopoUtil::get_bridge_adjacencies(), moab::Interface::get_entities_by_dimension(), moab::Interface::get_entities_by_handle(), interfaceSets, is_iface_proc(), MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, moab::Range::merge(), PSTATUS_NOT, PSTATUS_NOT_OWNED, moab::Range::subset_by_dimension(), and moab::Interface::UNION.

Referenced by get_sent_ents().

◆ get_global_part_count()

ErrorCode moab::ParallelComm::get_global_part_count ( int &  count_out) const

Definition at line 8506 of file ParallelComm.cpp.

8507 {
8508  count_out = globalPartCount;
8509  return count_out < 0 ? MB_FAILURE : MB_SUCCESS;
8510 }

References globalPartCount, and MB_SUCCESS.

◆ get_id()

int moab::ParallelComm::get_id ( ) const
inline

Get ID used to reference this PCOMM instance.

Definition at line 70 of file ParallelComm.hpp.

71  {
72  return pcommID;
73  }

References pcommID.

Referenced by iMOAB_LoadMesh(), internal_WriteMesh(), and DeformMeshRemap::read_file().

◆ get_iface_entities()

ErrorCode moab::ParallelComm::get_iface_entities ( int  other_proc,
int  dim,
Range &  iface_ents 
)

Get entities on interfaces shared with another proc.

Parameters
other_procOther proc sharing the interface
dimDimension of entities to return, -1 if all dims
iface_entsReturned entities

Definition at line 7546 of file ParallelComm.cpp.

7547 {
7548  Range iface_sets;
7549  ErrorCode result = MB_SUCCESS;
7550 
7551  for( Range::iterator rit = interfaceSets.begin(); rit != interfaceSets.end(); ++rit )
7552  {
7553  if( -1 != other_proc && !is_iface_proc( *rit, other_proc ) ) continue;
7554 
7555  if( -1 == dim )
7556  {
7557  result = mbImpl->get_entities_by_handle( *rit, iface_ents );
7558  MB_CHK_SET_ERR( result, "Failed to get entities in iface set" );
7559  }
7560  else
7561  {
7562  result = mbImpl->get_entities_by_dimension( *rit, dim, iface_ents );
7563  MB_CHK_SET_ERR( result, "Failed to get entities in iface set" );
7564  }
7565  }
7566 
7567  return MB_SUCCESS;
7568 }

References moab::Range::begin(), moab::Range::end(), ErrorCode, moab::Interface::get_entities_by_dimension(), moab::Interface::get_entities_by_handle(), interfaceSets, is_iface_proc(), MB_CHK_SET_ERR, MB_SUCCESS, and mbImpl.

Referenced by get_sent_ents().

◆ get_interface_procs()

ErrorCode moab::ParallelComm::get_interface_procs ( std::set< unsigned int > &  iface_procs,
const bool  get_buffs = false 
)

get processors with which this processor shares an interface

Get processors with which this processor communicates; sets are sorted by processor.

Definition at line 5633 of file ParallelComm.cpp.

5634 {
5635  // Make sure the sharing procs vector is empty
5636  procs_set.clear();
5637 
5638  // Pre-load vector of single-proc tag values
5639  unsigned int i, j;
5640  std::vector< int > iface_proc( interfaceSets.size() );
5641  ErrorCode result = mbImpl->tag_get_data( sharedp_tag(), interfaceSets, iface_proc.data() );
5642  MB_CHK_SET_ERR( result, "Failed to get iface_proc for iface sets" );
5643 
5644  // Get sharing procs either from single-proc vector or by getting
5645  // multi-proc tag value
5646  int tmp_iface_procs[MAX_SHARING_PROCS];
5647  std::fill( tmp_iface_procs, tmp_iface_procs + MAX_SHARING_PROCS, -1 );
5648  Range::iterator rit;
5649  for( rit = interfaceSets.begin(), i = 0; rit != interfaceSets.end(); ++rit, i++ )
5650  {
5651  if( -1 != iface_proc[i] )
5652  {
5653  assert( iface_proc[i] != (int)procConfig.proc_rank() );
5654  procs_set.insert( (unsigned int)iface_proc[i] );
5655  }
5656  else
5657  {
5658  // Get the sharing_procs tag
5659  result = mbImpl->tag_get_data( sharedps_tag(), &( *rit ), 1, tmp_iface_procs );
5660  MB_CHK_SET_ERR( result, "Failed to get iface_procs for iface set" );
5661  for( j = 0; j < MAX_SHARING_PROCS; j++ )
5662  {
5663  if( -1 != tmp_iface_procs[j] && tmp_iface_procs[j] != (int)procConfig.proc_rank() )
5664  procs_set.insert( (unsigned int)tmp_iface_procs[j] );
5665  else if( -1 == tmp_iface_procs[j] )
5666  {
5667  std::fill( tmp_iface_procs, tmp_iface_procs + j, -1 );
5668  break;
5669  }
5670  }
5671  }
5672  }
5673 
5674  if( get_buffs )
5675  {
5676  for( std::set< unsigned int >::iterator sit = procs_set.begin(); sit != procs_set.end(); ++sit )
5677  get_buffers( *sit );
5678  }
5679 
5680  return MB_SUCCESS;
5681 }

References moab::Range::begin(), moab::Range::end(), ErrorCode, get_buffers(), interfaceSets, MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, moab::ProcConfig::proc_rank(), procConfig, sharedp_tag(), sharedps_tag(), moab::Range::size(), and moab::Interface::tag_get_data().

Referenced by get_comm_procs(), resolve_shared_ents(), and moab::ParallelMergeMesh::TagSharedElements().

◆ get_interface_sets()

ErrorCode moab::ParallelComm::get_interface_sets ( EntityHandle  part,
Range &  iface_sets_out,
int *  adj_part_id = 0 
)

Definition at line 8634 of file ParallelComm.cpp.

8635 {
8636  // FIXME : assumes one part per processor.
8637  // Need to store part iface sets as children to implement
8638  // this correctly.
8639  iface_sets_out = interface_sets();
8640 
8641  if( adj_part_id )
8642  {
8643  int part_ids[MAX_SHARING_PROCS], num_parts;
8644  Range::iterator i = iface_sets_out.begin();
8645  while( i != iface_sets_out.end() )
8646  {
8647  unsigned char pstat;
8648  ErrorCode rval = get_sharing_data( *i, part_ids, NULL, pstat, num_parts );
8649  if( MB_SUCCESS != rval ) return rval;
8650 
8651  if( std::find( part_ids, part_ids + num_parts, *adj_part_id ) - part_ids != num_parts )
8652  ++i;
8653  else
8654  i = iface_sets_out.erase( i );
8655  }
8656  }
8657 
8658  return MB_SUCCESS;
8659 }

References moab::Range::begin(), moab::Range::end(), moab::Range::erase(), ErrorCode, get_sharing_data(), interface_sets(), MAX_SHARING_PROCS, and MB_SUCCESS.

Referenced by get_part_neighbor_ids().

◆ get_local_handles() [1/3]

ErrorCode moab::ParallelComm::get_local_handles ( const Range &  remote_handles,
Range &  local_handles,
const std::vector< EntityHandle > &  new_ents 
)
private

same as above except puts results in range

Definition at line 3178 of file ParallelComm.cpp.

3181 {
3182  std::vector< EntityHandle > rh_vec;
3183  rh_vec.reserve( remote_handles.size() );
3184  std::copy( remote_handles.begin(), remote_handles.end(), std::back_inserter( rh_vec ) );
3185  ErrorCode result = get_local_handles( rh_vec.data(), remote_handles.size(), new_ents );
3186  std::copy( rh_vec.begin(), rh_vec.end(), range_inserter( local_handles ) );
3187  return result;
3188 }

References moab::Range::begin(), moab::Range::end(), ErrorCode, get_local_handles(), and moab::Range::size().

◆ get_local_handles() [2/3]

ErrorCode moab::ParallelComm::get_local_handles ( EntityHandle *  from_vec,
int  num_ents,
const Range &  new_ents 
)
private

goes through from_vec, and for any with type MBMAXTYPE, replaces with new_ents value at index corresponding to id of entity in from_vec

Definition at line 3190 of file ParallelComm.cpp.

3191 {
3192  std::vector< EntityHandle > tmp_ents;
3193  std::copy( new_ents.begin(), new_ents.end(), std::back_inserter( tmp_ents ) );
3194  return get_local_handles( from_vec, num_ents, tmp_ents );
3195 }

References moab::Range::begin(), and moab::Range::end().

Referenced by get_local_handles(), unpack_entities(), unpack_sets(), and unpack_tags().

◆ get_local_handles() [3/3]

ErrorCode moab::ParallelComm::get_local_handles ( EntityHandle *  from_vec,
int  num_ents,
const std::vector< EntityHandle > &  new_ents 
)
private

same as above except gets new_ents from vector

Definition at line 3197 of file ParallelComm.cpp.

3200 {
3201  for( int i = 0; i < num_ents; i++ )
3202  {
3203  if( TYPE_FROM_HANDLE( from_vec[i] ) == MBMAXTYPE )
3204  {
3205  assert( ID_FROM_HANDLE( from_vec[i] ) < (int)new_ents.size() );
3206  from_vec[i] = new_ents[ID_FROM_HANDLE( from_vec[i] )];
3207  }
3208  }
3209 
3210  return MB_SUCCESS;
3211 }

References moab::ID_FROM_HANDLE(), MB_SUCCESS, MBMAXTYPE, and moab::TYPE_FROM_HANDLE().

◆ get_moab()

Interface* moab::ParallelComm::get_moab ( ) const
inline

Definition at line 779 of file ParallelComm.hpp.

780  {
781  return mbImpl;
782  }

References mbImpl.

Referenced by moab::ParCommGraph::compute_partition(), exchange_ghost_cells(), moab::ParallelMergeMesh::ParallelMergeMesh(), moab::ParCommGraph::receive_mesh(), moab::ParCommGraph::receive_tag_values(), resolve_shared_ents(), moab::ParCommGraph::send_mesh_parts(), and moab::ParCommGraph::send_tag_values().

◆ get_owned_sets()

ErrorCode moab::ParallelComm::get_owned_sets ( unsigned  owning_rank,
Range &  sets_out 
) const

Get shared sets owned by process with specified rank.

Definition at line 9243 of file ParallelComm.cpp.

9244 {
9245  return sharedSetData->get_shared_sets( owning_rank, sets_out );
9246 }

References moab::SharedSetData::get_shared_sets(), and sharedSetData.

Referenced by moab::WriteHDF5Parallel::communicate_shared_set_ids(), and moab::WriteHDF5Parallel::create_meshset_tables().

◆ get_owner()

ErrorCode moab::ParallelComm::get_owner ( EntityHandle  entity,
int &  owner 
)
inline

Return the rank of the entity owner.

Definition at line 1643 of file ParallelComm.hpp.

1644 {
1645  EntityHandle tmp_handle;
1646  return get_owner_handle( entity, owner, tmp_handle );
1647 }

References get_owner_handle().

Referenced by moab::WriteHDF5Parallel::exchange_file_ids(), iMOAB_GetElementOwnership(), iMOAB_GetVertexOwnership(), iMOAB_GetVisibleElementsInfo(), and pack_shared_handles().

◆ get_owner_handle()

ErrorCode moab::ParallelComm::get_owner_handle ( EntityHandle  entity,
int &  owner,
EntityHandle &  handle 
)

Return the owner processor and handle of a given entity.

Return the rank of the entity owner.

Definition at line 8466 of file ParallelComm.cpp.

8467 {
8468  unsigned char pstat;
8469  int sharing_procs[MAX_SHARING_PROCS];
8470  EntityHandle sharing_handles[MAX_SHARING_PROCS];
8471 
8472  ErrorCode result = mbImpl->tag_get_data( pstatus_tag(), &entity, 1, &pstat );
8473  MB_CHK_SET_ERR( result, "Failed to get pstatus tag data" );
8474  if( !( pstat & PSTATUS_NOT_OWNED ) )
8475  {
8476  owner = proc_config().proc_rank();
8477  handle = entity;
8478  }
8479  else if( pstat & PSTATUS_MULTISHARED )
8480  {
8481  result = mbImpl->tag_get_data( sharedps_tag(), &entity, 1, sharing_procs );
8482  MB_CHK_SET_ERR( result, "Failed to get sharedps tag data" );
8483  owner = sharing_procs[0];
8484  result = mbImpl->tag_get_data( sharedhs_tag(), &entity, 1, sharing_handles );
8485  MB_CHK_SET_ERR( result, "Failed to get sharedhs tag data" );
8486  handle = sharing_handles[0];
8487  }
8488  else if( pstat & PSTATUS_SHARED )
8489  {
8490  result = mbImpl->tag_get_data( sharedp_tag(), &entity, 1, sharing_procs );
8491  MB_CHK_SET_ERR( result, "Failed to get sharedp tag data" );
8492  owner = sharing_procs[0];
8493  result = mbImpl->tag_get_data( sharedh_tag(), &entity, 1, sharing_handles );
8494  MB_CHK_SET_ERR( result, "Failed to get sharedh tag data" );
8495  handle = sharing_handles[0];
8496  }
8497  else
8498  {
8499  owner = -1;
8500  handle = 0;
8501  }
8502 
8503  return MB_SUCCESS;
8504 }

References ErrorCode, MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, proc_config(), moab::ProcConfig::proc_rank(), PSTATUS_MULTISHARED, PSTATUS_NOT_OWNED, PSTATUS_SHARED, pstatus_tag(), sharedh_tag(), sharedhs_tag(), sharedp_tag(), sharedps_tag(), and moab::Interface::tag_get_data().

Referenced by get_owner().

◆ get_owning_part()

ErrorCode moab::ParallelComm::get_owning_part ( EntityHandle  entity,
int &  owning_part_id_out,
EntityHandle *  owning_handle = 0 
)

Definition at line 8661 of file ParallelComm.cpp.

8662 {
8663  // FIXME : assumes one part per proc, and therefore part_id == rank
8664 
8665  // If entity is not shared, then we're the owner.
8666  unsigned char pstat;
8667  ErrorCode result = mbImpl->tag_get_data( pstatus_tag(), &handle, 1, &pstat );
8668  MB_CHK_SET_ERR( result, "Failed to get pstatus tag data" );
8669  if( !( pstat & PSTATUS_NOT_OWNED ) )
8670  {
8671  owning_part_id = proc_config().proc_rank();
8672  if( remote_handle ) *remote_handle = handle;
8673  return MB_SUCCESS;
8674  }
8675 
8676  // If entity is shared with one other proc, then
8677  // sharedp_tag will contain a positive value.
8678  result = mbImpl->tag_get_data( sharedp_tag(), &handle, 1, &owning_part_id );
8679  MB_CHK_SET_ERR( result, "Failed to get sharedp tag data" );
8680  if( owning_part_id != -1 )
8681  {
8682  // Done?
8683  if( !remote_handle ) return MB_SUCCESS;
8684 
8685  // Get handles on remote processors (and this one)
8686  return mbImpl->tag_get_data( sharedh_tag(), &handle, 1, remote_handle );
8687  }
8688 
8689  // If here, then the entity is shared with at least two other processors.
8690  // Get the list from the sharedps_tag
8691  const void* part_id_list = 0;
8692  result = mbImpl->tag_get_by_ptr( sharedps_tag(), &handle, 1, &part_id_list );
8693  if( MB_SUCCESS != result ) return result;
8694  owning_part_id = ( (const int*)part_id_list )[0];
8695 
8696  // Done?
8697  if( !remote_handle ) return MB_SUCCESS;
8698 
8699  // Get remote handles
8700  const void* handle_list = 0;
8701  result = mbImpl->tag_get_by_ptr( sharedhs_tag(), &handle, 1, &handle_list );
8702  if( MB_SUCCESS != result ) return result;
8703 
8704  *remote_handle = ( (const EntityHandle*)handle_list )[0];
8705  return MB_SUCCESS;
8706 }

References ErrorCode, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, proc_config(), moab::ProcConfig::proc_rank(), PSTATUS_NOT_OWNED, pstatus_tag(), sharedh_tag(), sharedhs_tag(), sharedp_tag(), sharedps_tag(), moab::Interface::tag_get_by_ptr(), and moab::Interface::tag_get_data().

◆ get_part_entities()

ErrorCode moab::ParallelComm::get_part_entities ( Range &  ents,
int  dim = -1 
)

return all the entities in parts owned locally

Definition at line 8445 of file ParallelComm.cpp.

8446 {
8447  ErrorCode result;
8448 
8449  for( Range::iterator rit = partitionSets.begin(); rit != partitionSets.end(); ++rit )
8450  {
8451  Range tmp_ents;
8452  if( -1 == dim )
8453  result = mbImpl->get_entities_by_handle( *rit, tmp_ents, true );
8454  else
8455  result = mbImpl->get_entities_by_dimension( *rit, dim, tmp_ents, true );
8456 
8457  if( MB_SUCCESS != result ) return result;
8458  ents.merge( tmp_ents );
8459  }
8460 
8461  return MB_SUCCESS;
8462 }

References moab::Range::begin(), moab::Range::end(), ErrorCode, moab::Interface::get_entities_by_dimension(), moab::Interface::get_entities_by_handle(), MB_SUCCESS, mbImpl, moab::Range::merge(), and partitionSets.

Referenced by moab::Coupler::initialize_tree(), and main().

◆ get_part_handle()

ErrorCode moab::ParallelComm::get_part_handle ( int  id,
EntityHandle &  handle_out 
) const

Definition at line 8526 of file ParallelComm.cpp.

8527 {
8528  // FIXME: assumes only 1 local part
8529  if( (unsigned)id != proc_config().proc_rank() ) return MB_ENTITY_NOT_FOUND;
8530  handle_out = partition_sets().front();
8531  return MB_SUCCESS;
8532 }

References moab::Range::front(), MB_ENTITY_NOT_FOUND, MB_SUCCESS, partition_sets(), and proc_config().

Referenced by assign_entities_part(), and remove_entities_part().

◆ get_part_id()

ErrorCode moab::ParallelComm::get_part_id ( EntityHandle  part,
int &  id_out 
) const

Definition at line 8519 of file ParallelComm.cpp.

8520 {
8521  // FIXME: assumes only 1 local part
8522  id_out = proc_config().proc_rank();
8523  return MB_SUCCESS;
8524 }

References MB_SUCCESS, proc_config(), and moab::ProcConfig::proc_rank().

Referenced by get_part_neighbor_ids().

◆ get_part_neighbor_ids()

ErrorCode moab::ParallelComm::get_part_neighbor_ids ( EntityHandle  part,
int  neighbors_out[MAX_SHARING_PROCS],
int &  num_neighbors_out 
)

Definition at line 8600 of file ParallelComm.cpp.

8603 {
8604  ErrorCode rval;
8605  Range iface;
8606  rval = get_interface_sets( part, iface );
8607  if( MB_SUCCESS != rval ) return rval;
8608 
8609  num_neighbors_out = 0;
8610  int n, j = 0;
8611  int tmp[MAX_SHARING_PROCS] = { 0 }, curr[MAX_SHARING_PROCS] = { 0 };
8612  int* parts[2] = { neighbors_out, tmp };
8613  for( Range::iterator i = iface.begin(); i != iface.end(); ++i )
8614  {
8615  unsigned char pstat;
8616  rval = get_sharing_data( *i, curr, NULL, pstat, n );
8617  if( MB_SUCCESS != rval ) return rval;
8618  std::sort( curr, curr + n );
8619  assert( num_neighbors_out < MAX_SHARING_PROCS );
8620  int* k = std::set_union( parts[j], parts[j] + num_neighbors_out, curr, curr + n, parts[1 - j] );
8621  j = 1 - j;
8622  num_neighbors_out = k - parts[j];
8623  }
8624  if( parts[j] != neighbors_out ) std::copy( parts[j], parts[j] + num_neighbors_out, neighbors_out );
8625 
8626  // Remove input part from list
8627  int id;
8628  rval = get_part_id( part, id );
8629  if( MB_SUCCESS == rval )
8630  num_neighbors_out = std::remove( neighbors_out, neighbors_out + num_neighbors_out, id ) - neighbors_out;
8631  return rval;
8632 }

References ErrorCode, get_interface_sets(), get_part_id(), get_sharing_data(), iface, MAX_SHARING_PROCS, and MB_SUCCESS.

◆ get_part_owner()

ErrorCode moab::ParallelComm::get_part_owner ( int  part_id,
int &  owner_out 
) const

Definition at line 8512 of file ParallelComm.cpp.

8513 {
8514  // FIXME: assumes only 1 local part
8515  owner = part_id;
8516  return MB_SUCCESS;
8517 }

References MB_SUCCESS.

◆ get_partitioning()

EntityHandle moab::ParallelComm::get_partitioning ( ) const
inline

Definition at line 725 of file ParallelComm.hpp.

726  {
727  return partitioningSet;
728  }

References partitioningSet.

Referenced by create_part(), and destroy_part().

◆ get_pcomm() [1/2]

ParallelComm * moab::ParallelComm::get_pcomm ( Interface *  impl,
const int  index 
)
static

get the indexed pcomm object from the interface

Examples
ExchangeHalos.cpp, GenLargeMesh.cpp, and ReduceExchangeTags.cpp.

Definition at line 8329 of file ParallelComm.cpp.

8330 {
8331  Tag pc_tag = pcomm_tag( impl, false );
8332  if( 0 == pc_tag ) return NULL;
8333 
8334  const EntityHandle root = 0;
8335  ParallelComm* pc_array[MAX_SHARING_PROCS];
8336  ErrorCode result = impl->tag_get_data( pc_tag, &root, 1, (void*)pc_array );
8337  if( MB_SUCCESS != result ) return NULL;
8338 
8339  return pc_array[index];
8340 }

References ErrorCode, moab::index, MAX_SHARING_PROCS, MB_SUCCESS, pcomm_tag(), and moab::Interface::tag_get_data().

Referenced by moab::ScdInterface::construct_box(), get_max_volume(), get_pcomm(), moab::HiReconstruction::HiReconstruction(), moab::Core::load_file(), main(), moab::MeshGeneration::MeshGeneration(), moab::NestedRefine::NestedRefine(), moab::WriteHDF5Parallel::parallel_create_file(), moab::ReadNC::parse_options(), moab::WriteNC::parse_options(), perform_laplacian_smoothing(), perform_lloyd_relaxation(), moab::ReadParallel::ReadParallel(), RuntimeContext::RuntimeContext(), and moab::ReadHDF5::set_up_read().

◆ get_pcomm() [2/2]

ParallelComm * moab::ParallelComm::get_pcomm ( Interface *  impl,
EntityHandle  partitioning,
const MPI_Comm *  comm = 0 
)
static

Get ParallelComm instance associated with partition handle Will create ParallelComm instance if a) one does not already exist and b) a valid value for MPI_Comm is passed.

get the indexed pcomm object from the interface

Definition at line 8361 of file ParallelComm.cpp.

8362 {
8363  ErrorCode rval;
8364  ParallelComm* result = 0;
8365 
8366  Tag prtn_tag;
8367  rval =
8368  impl->tag_get_handle( PARTITIONING_PCOMM_TAG_NAME, 1, MB_TYPE_INTEGER, prtn_tag, MB_TAG_SPARSE | MB_TAG_CREAT );
8369  if( MB_SUCCESS != rval ) return 0;
8370 
8371  int pcomm_id;
8372  rval = impl->tag_get_data( prtn_tag, &prtn, 1, &pcomm_id );
8373  if( MB_SUCCESS == rval )
8374  {
8375  result = get_pcomm( impl, pcomm_id );
8376  }
8377  else if( MB_TAG_NOT_FOUND == rval && comm )
8378  {
8379  result = new ParallelComm( impl, *comm, &pcomm_id );
8380  if( !result ) return 0;
8381  result->set_partitioning( prtn );
8382 
8383  rval = impl->tag_set_data( prtn_tag, &prtn, 1, &pcomm_id );
8384  if( MB_SUCCESS != rval )
8385  {
8386  delete result;
8387  result = 0;
8388  }
8389  }
8390 
8391  return result;
8392 }

References comm(), ErrorCode, get_pcomm(), MB_SUCCESS, MB_TAG_CREAT, MB_TAG_NOT_FOUND, MB_TAG_SPARSE, MB_TYPE_INTEGER, ParallelComm(), moab::PARTITIONING_PCOMM_TAG_NAME, set_partitioning(), moab::Interface::tag_get_data(), moab::Interface::tag_get_handle(), and moab::Interface::tag_set_data().

◆ get_proc_nvecs()

ErrorCode moab::ParallelComm::get_proc_nvecs ( int  resolve_dim,
int  shared_dim,
Range *  skin_ents,
std::map< std::vector< int >, std::vector< EntityHandle > > &  proc_nvecs 
)
private

Definition at line 5333 of file ParallelComm.cpp.

5337 {
5338  // Set sharing procs tags on other skin ents
5339  ErrorCode result;
5340  const EntityHandle* connect;
5341  int num_connect;
5342  std::set< int > sharing_procs;
5343  std::vector< EntityHandle > dum_connect;
5344  std::vector< int > sp_vec;
5345 
5346  for( int d = 3; d > 0; d-- )
5347  {
5348  if( resolve_dim == d ) continue;
5349 
5350  for( Range::iterator rit = skin_ents[d].begin(); rit != skin_ents[d].end(); ++rit )
5351  {
5352  // Get connectivity
5353  result = mbImpl->get_connectivity( *rit, connect, num_connect, false, &dum_connect );
5354  MB_CHK_SET_ERR( result, "Failed to get connectivity on non-vertex skin entities" );
5355 
5356  int op = ( resolve_dim < shared_dim ? Interface::UNION : Interface::INTERSECT );
5357  result = get_sharing_data( connect, num_connect, sharing_procs, op );
5358  MB_CHK_SET_ERR( result, "Failed to get sharing data in get_proc_nvecs" );
5359  if( sharing_procs.empty() ||
5360  ( sharing_procs.size() == 1 && *sharing_procs.begin() == (int)procConfig.proc_rank() ) )
5361  continue;
5362 
5363  // Need to specify sharing data correctly for entities or they will
5364  // end up in a different interface set than corresponding vertices
5365  if( sharing_procs.size() == 2 )
5366  {
5367  std::set< int >::iterator it = sharing_procs.find( proc_config().proc_rank() );
5368  assert( it != sharing_procs.end() );
5369  sharing_procs.erase( it );
5370  }
5371 
5372  // Intersection is the owning proc(s) for this skin ent
5373  sp_vec.clear();
5374  std::copy( sharing_procs.begin(), sharing_procs.end(), std::back_inserter( sp_vec ) );
5375  assert( sp_vec.size() != 2 );
5376  proc_nvecs[sp_vec].push_back( *rit );
5377  }
5378  }
5379 
5380 #ifndef NDEBUG
5381  // Shouldn't be any repeated entities in any of the vectors in proc_nvecs
5382  for( std::map< std::vector< int >, std::vector< EntityHandle > >::iterator mit = proc_nvecs.begin();
5383  mit != proc_nvecs.end(); ++mit )
5384  {
5385  std::vector< EntityHandle > tmp_vec = ( mit->second );
5386  std::sort( tmp_vec.begin(), tmp_vec.end() );
5387  std::vector< EntityHandle >::iterator vit = std::unique( tmp_vec.begin(), tmp_vec.end() );
5388  assert( vit == tmp_vec.end() );
5389  }
5390 #endif
5391 
5392  return MB_SUCCESS;
5393 }

References moab::Range::end(), ErrorCode, moab::Interface::get_connectivity(), get_sharing_data(), moab::Interface::INTERSECT, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, proc_config(), moab::ProcConfig::proc_rank(), procConfig, and moab::Interface::UNION.

Referenced by create_interface_sets(), resolve_shared_ents(), and moab::ParallelMergeMesh::TagSharedElements().

◆ get_pstatus()

ErrorCode moab::ParallelComm::get_pstatus ( EntityHandle  entity,
unsigned char &  pstatus_val 
)

Get parallel status of an entity Returns the parallel status of an entity.

Parameters
entityThe entity being queried
pstatus_valParallel status of the entity

Definition at line 5683 of file ParallelComm.cpp.

5684 {
5685  ErrorCode result = mbImpl->tag_get_data( pstatus_tag(), &entity, 1, &pstatus_val );
5686  MB_CHK_SET_ERR( result, "Failed to get pastatus tag data" );
5687  return result;
5688 }

References ErrorCode, MB_CHK_SET_ERR, mbImpl, pstatus_tag(), and moab::Interface::tag_get_data().

Referenced by check_my_shared_handles().

◆ get_pstatus_entities()

ErrorCode moab::ParallelComm::get_pstatus_entities ( int  dim,
unsigned char  pstatus_val,
Range &  pstatus_ents 
)

Get entities with the given pstatus bit(s) set Returns any entities whose pstatus tag value v satisfies (v & pstatus_val)

Parameters
dimDimension of entities to be returned, or -1 if any
pstatus_valpstatus value of desired entities
pstatus_entsEntities returned from function

Definition at line 5690 of file ParallelComm.cpp.

5691 {
5692  Range ents;
5693  ErrorCode result;
5694 
5695  if( -1 == dim )
5696  {
5697  result = mbImpl->get_entities_by_handle( 0, ents );
5698  MB_CHK_SET_ERR( result, "Failed to get all entities" );
5699  }
5700  else
5701  {
5702  result = mbImpl->get_entities_by_dimension( 0, dim, ents );
5703  MB_CHK_SET_ERR( result, "Failed to get entities of dimension " << dim );
5704  }
5705 
5706  std::vector< unsigned char > pstatus( ents.size() );
5707  result = mbImpl->tag_get_data( pstatus_tag(), ents, pstatus.data() );
5708  MB_CHK_SET_ERR( result, "Failed to get pastatus tag data" );
5709  Range::iterator rit = ents.begin();
5710  int i = 0;
5711  if( pstatus_val )
5712  {
5713  for( ; rit != ents.end(); i++, ++rit )
5714  {
5715  if( pstatus[i] & pstatus_val && ( -1 == dim || mbImpl->dimension_from_handle( *rit ) == dim ) )
5716  pstatus_ents.insert( *rit );
5717  }
5718  }
5719  else
5720  {
5721  for( ; rit != ents.end(); i++, ++rit )
5722  {
5723  if( !pstatus[i] && ( -1 == dim || mbImpl->dimension_from_handle( *rit ) == dim ) )
5724  pstatus_ents.insert( *rit );
5725  }
5726  }
5727 
5728  return MB_SUCCESS;
5729 }

References moab::Range::begin(), moab::Interface::dimension_from_handle(), moab::Range::end(), ErrorCode, moab::Interface::get_entities_by_dimension(), moab::Interface::get_entities_by_handle(), moab::Range::insert(), MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, pstatus_tag(), moab::Range::size(), and moab::Interface::tag_get_data().

Referenced by main().

◆ get_remote_handles() [1/4]

ErrorCode moab::ParallelComm::get_remote_handles ( const bool  store_remote_handles,
const Range &  from_range,
EntityHandle *  to_vec,
int  to_proc,
const std::vector< EntityHandle > &  new_ents 
)
private

same as other version, except packs range into vector

Definition at line 2030 of file ParallelComm.cpp.

2035 {
2036  // NOTE: THIS IMPLEMENTATION IS JUST LIKE THE VECTOR-BASED VERSION, NO REUSE
2037  // AT THIS TIME, SO IF YOU FIX A BUG IN THIS VERSION, IT MAY BE IN THE
2038  // OTHER VERSION TOO!!!
2039  if( from_range.empty() ) return MB_SUCCESS;
2040 
2041  if( !store_remote_handles )
2042  {
2043  int err;
2044  // In this case, substitute position in new_ents list
2045  Range::iterator rit;
2046  unsigned int i;
2047  for( rit = from_range.begin(), i = 0; rit != from_range.end(); ++rit, i++ )
2048  {
2049  int ind = std::lower_bound( new_ents.begin(), new_ents.end(), *rit ) - new_ents.begin();
2050  assert( new_ents[ind] == *rit );
2051  to_vec[i] = CREATE_HANDLE( MBMAXTYPE, ind, err );
2052  assert( to_vec[i] != 0 && !err && -1 != ind );
2053  }
2054  }
2055  else
2056  {
2057  Tag shp_tag, shps_tag, shh_tag, shhs_tag, pstat_tag;
2058  ErrorCode result = get_shared_proc_tags( shp_tag, shps_tag, shh_tag, shhs_tag, pstat_tag );
2059  MB_CHK_SET_ERR( result, "Failed to get shared proc tags" );
2060 
2061  // Get single-proc destination handles and shared procs
2062  std::vector< int > sharing_procs( from_range.size() );
2063  result = mbImpl->tag_get_data( shh_tag, from_range, to_vec );
2064  MB_CHK_SET_ERR( result, "Failed to get shared handle tag for remote_handles" );
2065  result = mbImpl->tag_get_data( shp_tag, from_range, sharing_procs.data() );
2066  MB_CHK_SET_ERR( result, "Failed to get sharing proc tag in remote_handles" );
2067  for( unsigned int j = 0; j < from_range.size(); j++ )
2068  {
2069  if( to_vec[j] && sharing_procs[j] != to_proc ) to_vec[j] = 0;
2070  }
2071 
2072  EntityHandle tmp_handles[MAX_SHARING_PROCS];
2073  int tmp_procs[MAX_SHARING_PROCS];
2074  // Go through results, and for 0-valued ones, look for multiple shared proc
2075  Range::iterator rit;
2076  unsigned int i;
2077  for( rit = from_range.begin(), i = 0; rit != from_range.end(); ++rit, i++ )
2078  {
2079  if( !to_vec[i] )
2080  {
2081  result = mbImpl->tag_get_data( shhs_tag, &( *rit ), 1, tmp_handles );
2082  if( MB_SUCCESS == result )
2083  {
2084  result = mbImpl->tag_get_data( shps_tag, &( *rit ), 1, tmp_procs );
2085  MB_CHK_SET_ERR( result, "Failed to get sharedps tag data" );
2086  for( int j = 0; j < MAX_SHARING_PROCS; j++ )
2087  if( tmp_procs[j] == to_proc )
2088  {
2089  to_vec[i] = tmp_handles[j];
2090  break;
2091  }
2092  }
2093 
2094  if( !to_vec[i] )
2095  {
2096  int j = std::lower_bound( new_ents.begin(), new_ents.end(), *rit ) - new_ents.begin();
2097  if( (int)new_ents.size() == j )
2098  {
2099  MB_SET_ERR( MB_FAILURE, "Failed to find new entity in send list" );
2100  }
2101  int err;
2102  to_vec[i] = CREATE_HANDLE( MBMAXTYPE, j, err );
2103  if( err )
2104  {
2105  MB_SET_ERR( MB_FAILURE, "Failed to create handle in remote_handles" );
2106  }
2107  }
2108  }
2109  }
2110  }
2111 
2112  return MB_SUCCESS;
2113 }

References moab::Range::begin(), moab::CREATE_HANDLE(), moab::Range::empty(), moab::Range::end(), ErrorCode, get_shared_proc_tags(), MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SET_ERR, MB_SUCCESS, mbImpl, MBMAXTYPE, moab::Range::size(), and moab::Interface::tag_get_data().

◆ get_remote_handles() [2/4]

ErrorCode moab::ParallelComm::get_remote_handles ( const bool  store_remote_handles,
const Range &  from_range,
Range &  to_range,
int  to_proc,
const std::vector< EntityHandle > &  new_ents 
)
private

same as other version, except from_range and to_range should be different here

Definition at line 2115 of file ParallelComm.cpp.

2120 {
2121  std::vector< EntityHandle > to_vector( from_range.size() );
2122 
2123  ErrorCode result = get_remote_handles( store_remote_handles, from_range, to_vector.data(), to_proc, new_ents );
2124  MB_CHK_SET_ERR( result, "Failed to get remote handles" );
2125  std::copy( to_vector.begin(), to_vector.end(), range_inserter( to_range ) );
2126  return result;
2127 }

References ErrorCode, get_remote_handles(), MB_CHK_SET_ERR, and moab::Range::size().

◆ get_remote_handles() [3/4]

ErrorCode moab::ParallelComm::get_remote_handles ( const bool  store_remote_handles,
EntityHandle *  from_vec,
EntityHandle *  to_vec_tmp,
int  num_ents,
int  to_proc,
const std::vector< EntityHandle > &  new_ents 
)
private

replace handles in from_vec with corresponding handles on to_proc (by checking shared[p/h]_tag and shared[p/h]s_tag; if no remote handle and new_ents is non-null, substitute instead CREATE_HANDLE(MBMAXTYPE, index) where index is handle's position in new_ents

Definition at line 1927 of file ParallelComm.cpp.

1933 {
1934  // NOTE: THIS IMPLEMENTATION IS JUST LIKE THE RANGE-BASED VERSION, NO REUSE
1935  // AT THIS TIME, SO IF YOU FIX A BUG IN THIS VERSION, IT MAY BE IN THE
1936  // OTHER VERSION TOO!!!
1937  if( 0 == num_ents ) return MB_SUCCESS;
1938 
1939  // Use a local destination ptr in case we're doing an in-place copy
1940  std::vector< EntityHandle > tmp_vector;
1941  EntityHandle* to_vec = to_vec_tmp;
1942  if( to_vec == from_vec )
1943  {
1944  tmp_vector.resize( num_ents );
1945  to_vec = tmp_vector.data();
1946  }
1947 
1948  if( !store_remote_handles )
1949  {
1950  int err;
1951  // In this case, substitute position in new_ents list
1952  for( int i = 0; i < num_ents; i++ )
1953  {
1954  int ind = std::lower_bound( new_ents.begin(), new_ents.end(), from_vec[i] ) - new_ents.begin();
1955  assert( new_ents[ind] == from_vec[i] );
1956  to_vec[i] = CREATE_HANDLE( MBMAXTYPE, ind, err );
1957  assert( to_vec[i] != 0 && !err && -1 != ind );
1958  }
1959  }
1960  else
1961  {
1962  Tag shp_tag, shps_tag, shh_tag, shhs_tag, pstat_tag;
1963  ErrorCode result = get_shared_proc_tags( shp_tag, shps_tag, shh_tag, shhs_tag, pstat_tag );
1964  MB_CHK_SET_ERR( result, "Failed to get shared proc tags" );
1965 
1966  // Get single-proc destination handles and shared procs
1967  std::vector< int > sharing_procs( num_ents );
1968  result = mbImpl->tag_get_data( shh_tag, from_vec, num_ents, to_vec );
1969  MB_CHK_SET_ERR( result, "Failed to get shared handle tag for remote_handles" );
1970  result = mbImpl->tag_get_data( shp_tag, from_vec, num_ents, sharing_procs.data() );
1971  MB_CHK_SET_ERR( result, "Failed to get sharing proc tag in remote_handles" );
1972  for( int j = 0; j < num_ents; j++ )
1973  {
1974  if( to_vec[j] && sharing_procs[j] != to_proc ) to_vec[j] = 0;
1975  }
1976 
1977  EntityHandle tmp_handles[MAX_SHARING_PROCS];
1978  int tmp_procs[MAX_SHARING_PROCS];
1979  int i;
1980  // Go through results, and for 0-valued ones, look for multiple shared proc
1981  for( i = 0; i < num_ents; i++ )
1982  {
1983  if( !to_vec[i] )
1984  {
1985  result = mbImpl->tag_get_data( shps_tag, from_vec + i, 1, tmp_procs );
1986  if( MB_SUCCESS == result )
1987  {
1988  for( int j = 0; j < MAX_SHARING_PROCS; j++ )
1989  {
1990  if( -1 == tmp_procs[j] )
1991  break;
1992  else if( tmp_procs[j] == to_proc )
1993  {
1994  result = mbImpl->tag_get_data( shhs_tag, from_vec + i, 1, tmp_handles );
1995  MB_CHK_SET_ERR( result, "Failed to get sharedhs tag data" );
1996  to_vec[i] = tmp_handles[j];
1997  assert( to_vec[i] );
1998  break;
1999  }
2000  }
2001  }
2002  if( !to_vec[i] )
2003  {
2004  int j = std::lower_bound( new_ents.begin(), new_ents.end(), from_vec[i] ) - new_ents.begin();
2005  if( (int)new_ents.size() == j )
2006  {
2007  std::cout << "Failed to find new entity in send list, proc " << procConfig.proc_rank()
2008  << std::endl;
2009  for( int k = 0; k <= num_ents; k++ )
2010  std::cout << k << ": " << from_vec[k] << " " << to_vec[k] << std::endl;
2011  MB_SET_ERR( MB_FAILURE, "Failed to find new entity in send list" );
2012  }
2013  int err;
2014  to_vec[i] = CREATE_HANDLE( MBMAXTYPE, j, err );
2015  if( err )
2016  {
2017  MB_SET_ERR( MB_FAILURE, "Failed to create handle in remote_handles" );
2018  }
2019  }
2020  }
2021  }
2022  }
2023 
2024  // memcpy over results if from_vec and to_vec are the same
2025  if( to_vec_tmp == from_vec ) memcpy( from_vec, to_vec, num_ents * sizeof( EntityHandle ) );
2026 
2027  return MB_SUCCESS;
2028 }

References moab::CREATE_HANDLE(), ErrorCode, get_shared_proc_tags(), MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SET_ERR, MB_SUCCESS, mbImpl, MBMAXTYPE, moab::ProcConfig::proc_rank(), procConfig, and moab::Interface::tag_get_data().

◆ get_remote_handles() [4/4]

ErrorCode moab::ParallelComm::get_remote_handles ( EntityHandle *  local_vec,
EntityHandle *  rem_vec,
int  num_ents,
int  to_proc 
)

Definition at line 1081 of file ParallelComm.cpp.

1082 {
1083  ErrorCode error;
1084  std::vector< EntityHandle > newents;
1085  error = get_remote_handles( true, local_vec, rem_vec, num_ents, to_proc, newents );
1086  MB_CHK_ERR( error );
1087 
1088  return MB_SUCCESS;
1089 }

References moab::error(), ErrorCode, MB_CHK_ERR, and MB_SUCCESS.

Referenced by check_my_shared_handles(), get_remote_handles(), pack_entity_seq(), pack_sets(), pack_tag(), and settle_intersection_points().

◆ get_sent_ents()

ErrorCode moab::ParallelComm::get_sent_ents ( const bool  is_iface,
const int  bridge_dim,
const int  ghost_dim,
const int  num_layers,
const int  addl_ents,
Range *  sent_ents,
Range &  allsent,
TupleList &  entprocs 
)
private

Definition at line 6759 of file ParallelComm.cpp.

6767 {
6768  ErrorCode result;
6769  unsigned int ind;
6770  std::vector< unsigned int >::iterator proc_it;
6771  Range tmp_range;
6772 
6773  // Done in a separate loop over procs because sometimes later procs
6774  // need to add info to earlier procs' messages
6775  for( ind = 0, proc_it = buffProcs.begin(); proc_it != buffProcs.end(); ++proc_it, ind++ )
6776  {
6777  if( !is_iface )
6778  {
6779  result =
6780  get_ghosted_entities( bridge_dim, ghost_dim, buffProcs[ind], num_layers, addl_ents, sent_ents[ind] );
6781  MB_CHK_SET_ERR( result, "Failed to get ghost layers" );
6782  }
6783  else
6784  {
6785  result = get_iface_entities( buffProcs[ind], -1, sent_ents[ind] );
6786  MB_CHK_SET_ERR( result, "Failed to get interface layers" );
6787  }
6788 
6789  // Filter out entities already shared with destination
6790  tmp_range.clear();
6791  result = filter_pstatus( sent_ents[ind], PSTATUS_SHARED, PSTATUS_AND, buffProcs[ind], &tmp_range );
6792  MB_CHK_SET_ERR( result, "Failed to filter on owner" );
6793  if( !tmp_range.empty() ) sent_ents[ind] = subtract( sent_ents[ind], tmp_range );
6794 
6795  allsent.merge( sent_ents[ind] );
6796  }
6797 
6798  //===========================================
6799  // Need to get procs each entity is sent to
6800  //===========================================
6801 
6802  // Get the total # of proc/handle pairs
6803  int npairs = 0;
6804  for( ind = 0; ind < buffProcs.size(); ind++ )
6805  npairs += sent_ents[ind].size();
6806 
6807  // Allocate a TupleList of that size
6808  entprocs.initialize( 1, 0, 1, 0, npairs );
6809  entprocs.enableWriteAccess();
6810 
6811  // Put the proc/handle pairs in the list
6812  for( ind = 0, proc_it = buffProcs.begin(); proc_it != buffProcs.end(); ++proc_it, ind++ )
6813  {
6814  for( Range::iterator rit = sent_ents[ind].begin(); rit != sent_ents[ind].end(); ++rit )
6815  {
6816  entprocs.vi_wr[entprocs.get_n()] = *proc_it;
6817  entprocs.vul_wr[entprocs.get_n()] = *rit;
6818  entprocs.inc_n();
6819  }
6820  }
6821  // Sort by handle
6822  moab::TupleList::buffer sort_buffer;
6823  sort_buffer.buffer_init( npairs );
6824  entprocs.sort( 1, &sort_buffer );
6825 
6826  entprocs.disableWriteAccess();
6827  sort_buffer.reset();
6828 
6829  return MB_SUCCESS;
6830 }

References buffProcs, moab::Range::clear(), moab::TupleList::disableWriteAccess(), moab::Range::empty(), moab::TupleList::enableWriteAccess(), moab::Range::end(), ErrorCode, filter_pstatus(), get_ghosted_entities(), get_iface_entities(), moab::TupleList::get_n(), moab::TupleList::inc_n(), moab::TupleList::initialize(), MB_CHK_SET_ERR, MB_SUCCESS, moab::Range::merge(), PSTATUS_AND, PSTATUS_SHARED, moab::TupleList::buffer::reset(), size(), moab::TupleList::sort(), moab::subtract(), moab::TupleList::vi_wr, and moab::TupleList::vul_wr.

Referenced by exchange_ghost_cells().

◆ get_shared_entities()

ErrorCode moab::ParallelComm::get_shared_entities ( int  other_proc,
Range &  shared_ents,
int  dim = -1,
const bool  iface = false,
const bool  owned_filter = false 
)

Get shared entities of specified dimension If other_proc is -1, any shared entities are returned. If dim is -1, entities of all dimensions on interface are returned.

Parameters
other_procRank of processor for which interface entities are requested
shared_entsEntities returned from function
dimDimension of interface entities requested
ifaceIf true, return only entities on the interface
owned_filterIf true, return only owned shared entities

Definition at line 9129 of file ParallelComm.cpp.

9134 {
9135  shared_ents.clear();
9136  ErrorCode result = MB_SUCCESS;
9137 
9138  // Dimension
9139  if( -1 != dim )
9140  {
9141  DimensionPair dp = CN::TypeDimensionMap[dim];
9142  Range dum_range;
9143  std::copy( sharedEnts.begin(), sharedEnts.end(), range_inserter( dum_range ) );
9144  shared_ents.merge( dum_range.lower_bound( dp.first ), dum_range.upper_bound( dp.second ) );
9145  }
9146  else
9147  std::copy( sharedEnts.begin(), sharedEnts.end(), range_inserter( shared_ents ) );
9148 
9149  // Filter by iface
9150  if( iface )
9151  {
9152  result = filter_pstatus( shared_ents, PSTATUS_INTERFACE, PSTATUS_AND );
9153  MB_CHK_SET_ERR( result, "Failed to filter by iface" );
9154  }
9155 
9156  // Filter by owned
9157  if( owned_filter )
9158  {
9159  result = filter_pstatus( shared_ents, PSTATUS_NOT_OWNED, PSTATUS_NOT );
9160  MB_CHK_SET_ERR( result, "Failed to filter by owned" );
9161  }
9162 
9163  // Filter by proc
9164  if( -1 != other_proc )
9165  {
9166  result = filter_pstatus( shared_ents, PSTATUS_SHARED, PSTATUS_AND, other_proc );
9167  MB_CHK_SET_ERR( result, "Failed to filter by proc" );
9168  }
9169 
9170  return result;
9171 }

References moab::Range::clear(), ErrorCode, filter_pstatus(), iface, moab::Range::lower_bound(), MB_CHK_SET_ERR, MB_SUCCESS, moab::Range::merge(), PSTATUS_AND, PSTATUS_INTERFACE, PSTATUS_NOT, PSTATUS_NOT_OWNED, PSTATUS_SHARED, sharedEnts, moab::CN::TypeDimensionMap, and moab::Range::upper_bound().

Referenced by check_my_shared_handles(), moab::ParCommGraph::compute_partition(), perform_laplacian_smoothing(), and perform_lloyd_relaxation().

◆ get_shared_proc_tags()

ErrorCode moab::ParallelComm::get_shared_proc_tags ( Tag &  sharedp_tag,
Tag &  sharedps_tag,
Tag &  sharedh_tag,
Tag &  sharedhs_tag,
Tag &  pstatus_tag 
)
inline

return the tags used to indicate shared procs and handles

Definition at line 1574 of file ParallelComm.hpp.

1579 {
1580  sharedp = sharedp_tag();
1581  sharedps = sharedps_tag();
1582  sharedh = sharedh_tag();
1583  sharedhs = sharedhs_tag();
1584  pstatus = pstatus_tag();
1585 
1586  return MB_SUCCESS;
1587 }

References MB_SUCCESS, pstatus_tag(), sharedh_tag(), sharedhs_tag(), sharedp_tag(), and sharedps_tag().

Referenced by create_interface_sets(), get_remote_handles(), resolve_shared_ents(), and tag_shared_verts().

◆ get_shared_sets()

ErrorCode moab::ParallelComm::get_shared_sets ( Range &  result) const

Get all shared sets.

Definition at line 9233 of file ParallelComm.cpp.

9234 {
9235  return sharedSetData->get_shared_sets( result );
9236 }

References moab::SharedSetData::get_shared_sets(), and sharedSetData.

Referenced by moab::WriteHDF5Parallel::create_meshset_tables().

◆ get_sharing_data() [1/4]

ErrorCode moab::ParallelComm::get_sharing_data ( const EntityHandle *  entities,
int  num_entities,
std::set< int > &  procs,
int  op = Interface::INTERSECT 
)
inline

Get the intersection or union of all sharing processors Get the intersection or union of all sharing processors. Processor set is cleared as part of this function.

Parameters
entitiesEntity list ptr
num_entitiesNumber of entities
procsProcessors returned
opEither Interface::UNION or Interface::INTERSECT

Definition at line 1673 of file ParallelComm.hpp.

1677 {
1678  Range dum_range;
1679  // cast away constness 'cuz the range is passed as const
1680  EntityHandle* ents_cast = const_cast< EntityHandle* >( entities );
1681  std::copy( ents_cast, ents_cast + num_entities, range_inserter( dum_range ) );
1682  return get_sharing_data( dum_range, procs, op );
1683 }

References get_sharing_data().

◆ get_sharing_data() [2/4]

ErrorCode moab::ParallelComm::get_sharing_data ( const EntityHandle  entity,
int *  ps,
EntityHandle *  hs,
unsigned char &  pstat,
int &  num_ps 
)
inline

Get the shared processors/handles for an entity Same as other version but with int num_ps.

Parameters
entityEntity being queried
psPointer to sharing proc data
hsPointer to shared proc handle data
pstatReference to pstatus data returned from this function

Definition at line 1685 of file ParallelComm.hpp.

1690 {
1691  unsigned int dum_ps;
1692  ErrorCode result = get_sharing_data( entity, ps, hs, pstat, dum_ps );
1693  if( MB_SUCCESS == result ) num_ps = dum_ps;
1694  return result;
1695 }

References ErrorCode, get_sharing_data(), and MB_SUCCESS.

◆ get_sharing_data() [3/4]

ErrorCode moab::ParallelComm::get_sharing_data ( const EntityHandle  entity,
int *  ps,
EntityHandle *  hs,
unsigned char &  pstat,
unsigned int &  num_ps 
)

Get the shared processors/handles for an entity Get the shared processors/handles for an entity. Arrays must be large enough to receive data for all sharing procs. Does not include this proc if only shared with one other proc.

Parameters
entityEntity being queried
psPointer to sharing proc data
hsPointer to shared proc handle data
pstatReference to pstatus data returned from this function

Definition at line 3089 of file ParallelComm.cpp.

3094 {
3095  ErrorCode result = mbImpl->tag_get_data( pstatus_tag(), &entity, 1, &pstat );
3096  MB_CHK_SET_ERR( result, "Failed to get pstatus tag data" );
3097  if( pstat & PSTATUS_MULTISHARED )
3098  {
3099  result = mbImpl->tag_get_data( sharedps_tag(), &entity, 1, ps );
3100  MB_CHK_SET_ERR( result, "Failed to get sharedps tag data" );
3101  if( hs )
3102  {
3103  result = mbImpl->tag_get_data( sharedhs_tag(), &entity, 1, hs );
3104  MB_CHK_SET_ERR( result, "Failed to get sharedhs tag data" );
3105  }
3106  num_ps = std::find( ps, ps + MAX_SHARING_PROCS, -1 ) - ps;
3107  }
3108  else if( pstat & PSTATUS_SHARED )
3109  {
3110  result = mbImpl->tag_get_data( sharedp_tag(), &entity, 1, ps );
3111  MB_CHK_SET_ERR( result, "Failed to get sharedp tag data" );
3112  if( hs )
3113  {
3114  result = mbImpl->tag_get_data( sharedh_tag(), &entity, 1, hs );
3115  MB_CHK_SET_ERR( result, "Failed to get sharedh tag data" );
3116  hs[1] = 0;
3117  }
3118  // Initialize past end of data
3119  ps[1] = -1;
3120  num_ps = 1;
3121  }
3122  else
3123  {
3124  ps[0] = -1;
3125  if( hs ) hs[0] = 0;
3126  num_ps = 0;
3127  }
3128 
3129  assert( MAX_SHARING_PROCS >= num_ps );
3130 
3131  return MB_SUCCESS;
3132 }

References ErrorCode, MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, PSTATUS_MULTISHARED, PSTATUS_SHARED, pstatus_tag(), sharedh_tag(), sharedhs_tag(), sharedp_tag(), sharedps_tag(), and moab::Interface::tag_get_data().

Referenced by augment_default_sets_with_ghosts(), build_sharedhps_list(), check_clean_iface(), check_local_shared(), moab::ParCommGraph::compute_partition(), correct_thin_ghost_layers(), create_interface_sets(), delete_entities(), exchange_owned_meshs(), get_interface_sets(), get_part_neighbor_ids(), get_proc_nvecs(), get_sharing_data(), list_entities(), pack_shared_handles(), update_remote_data(), and update_remote_data_old().

◆ get_sharing_data() [4/4]

ErrorCode moab::ParallelComm::get_sharing_data ( const Range &  entities,
std::set< int > &  procs,
int  op = Interface::INTERSECT 
)

Get the intersection or union of all sharing processors Same as previous variant but with range as input.

Definition at line 3041 of file ParallelComm.cpp.

3042 {
3043  // Get the union or intersection of sharing data for multiple entities
3044  ErrorCode result;
3045  int sp2[MAX_SHARING_PROCS];
3046  int num_ps;
3047  unsigned char pstat;
3048  std::set< int > tmp_procs;
3049  procs.clear();
3050 
3051  for( Range::const_iterator rit = entities.begin(); rit != entities.end(); ++rit )
3052  {
3053  // Get sharing procs
3054  result = get_sharing_data( *rit, sp2, NULL, pstat, num_ps );
3055  MB_CHK_SET_ERR( result, "Failed to get sharing data in get_sharing_data" );
3056  if( !( pstat & PSTATUS_SHARED ) && Interface::INTERSECT == operation )
3057  {
3058  procs.clear();
3059  return MB_SUCCESS;
3060  }
3061 
3062  if( rit == entities.begin() )
3063  {
3064  std::copy( sp2, sp2 + num_ps, std::inserter( procs, procs.begin() ) );
3065  }
3066  else
3067  {
3068  std::sort( sp2, sp2 + num_ps );
3069  tmp_procs.clear();
3070  if( Interface::UNION == operation )
3071  std::set_union( procs.begin(), procs.end(), sp2, sp2 + num_ps,
3072  std::inserter( tmp_procs, tmp_procs.end() ) );
3073  else if( Interface::INTERSECT == operation )
3074  std::set_intersection( procs.begin(), procs.end(), sp2, sp2 + num_ps,
3075  std::inserter( tmp_procs, tmp_procs.end() ) );
3076  else
3077  {
3078  assert( "Unknown operation." && false );
3079  return MB_FAILURE;
3080  }
3081  procs.swap( tmp_procs );
3082  }
3083  if( Interface::INTERSECT == operation && procs.empty() ) return MB_SUCCESS;
3084  }
3085 
3086  return MB_SUCCESS;
3087 }

References moab::Range::begin(), moab::Range::end(), ErrorCode, get_sharing_data(), moab::Interface::INTERSECT, MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SUCCESS, PSTATUS_SHARED, and moab::Interface::UNION.

◆ get_sharing_parts()

ErrorCode moab::ParallelComm::get_sharing_parts ( EntityHandle  entity,
int  part_ids_out[MAX_SHARING_PROCS],
int &  num_part_ids_out,
EntityHandle  remote_handles[MAX_SHARING_PROCS] = 0 
)

Definition at line 8708 of file ParallelComm.cpp.

8712 {
8713  // FIXME : assumes one part per proc, and therefore part_id == rank
8714 
8715  // If entity is not shared, then we're the owner.
8716  unsigned char pstat;
8717  ErrorCode result = mbImpl->tag_get_data( pstatus_tag(), &entity, 1, &pstat );
8718  MB_CHK_SET_ERR( result, "Failed to get pstatus tag data" );
8719  if( !( pstat & PSTATUS_SHARED ) )
8720  {
8721  part_ids_out[0] = proc_config().proc_rank();
8722  if( remote_handles ) remote_handles[0] = entity;
8723  num_part_ids_out = 1;
8724  return MB_SUCCESS;
8725  }
8726 
8727  // If entity is shared with one other proc, then
8728  // sharedp_tag will contain a positive value.
8729  result = mbImpl->tag_get_data( sharedp_tag(), &entity, 1, part_ids_out );
8730  MB_CHK_SET_ERR( result, "Failed to get sharedp tag data" );
8731  if( part_ids_out[0] != -1 )
8732  {
8733  num_part_ids_out = 2;
8734  part_ids_out[1] = proc_config().proc_rank();
8735 
8736  // Done?
8737  if( !remote_handles ) return MB_SUCCESS;
8738 
8739  // Get handles on remote processors (and this one)
8740  remote_handles[1] = entity;
8741  return mbImpl->tag_get_data( sharedh_tag(), &entity, 1, remote_handles );
8742  }
8743 
8744  // If here, then the entity is shared with at least two other processors.
8745  // Get the list from the sharedps_tag
8746  result = mbImpl->tag_get_data( sharedps_tag(), &entity, 1, part_ids_out );
8747  if( MB_SUCCESS != result ) return result;
8748  // Count number of valid (positive) entries in sharedps_tag
8749  for( num_part_ids_out = 0; num_part_ids_out < MAX_SHARING_PROCS && part_ids_out[num_part_ids_out] >= 0;
8750  num_part_ids_out++ )
8751  ;
8752  // part_ids_out[num_part_ids_out++] = proc_config().proc_rank();
8753 #ifndef NDEBUG
8754  int my_idx = std::find( part_ids_out, part_ids_out + num_part_ids_out, proc_config().proc_rank() ) - part_ids_out;
8755  assert( my_idx < num_part_ids_out );
8756 #endif
8757 
8758  // Done?
8759  if( !remote_handles ) return MB_SUCCESS;
8760 
8761  // Get remote handles
8762  result = mbImpl->tag_get_data( sharedhs_tag(), &entity, 1, remote_handles );
8763  // remote_handles[num_part_ids_out - 1] = entity;
8764  assert( remote_handles[my_idx] == entity );
8765 
8766  return result;
8767 }

References ErrorCode, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, proc_config(), moab::ProcConfig::proc_rank(), PSTATUS_SHARED, pstatus_tag(), sharedh_tag(), sharedhs_tag(), sharedp_tag(), sharedps_tag(), and moab::Interface::tag_get_data().

◆ get_tag_send_list()

ErrorCode moab::ParallelComm::get_tag_send_list ( const Range &  all_entities,
std::vector< Tag > &  all_tags,
std::vector< Range > &  tag_ranges 
)
private

Get list of tags for which to exchange data.

Get tags and entities for which to exchange tag data. This function was originally part of 'pack_tags' requested with the 'all_possible_tags' parameter.

Parameters
all_entitiesInput. The set of entities for which data is to be communicated.
all_tagsOutput. Populated with the handles of tags to be sent.
tag_rangesOutput. For each corresponding tag in all_tags, the subset of 'all_entities' for which a tag value has been set.

Definition at line 3761 of file ParallelComm.cpp.

3764 {
3765  std::vector< Tag > tmp_tags;
3766  ErrorCode result = mbImpl->tag_get_tags( tmp_tags );
3767  MB_CHK_SET_ERR( result, "Failed to get tags in pack_tags" );
3768 
3769  std::vector< Tag >::iterator tag_it;
3770  for( tag_it = tmp_tags.begin(); tag_it != tmp_tags.end(); ++tag_it )
3771  {
3772  std::string tag_name;
3773  result = mbImpl->tag_get_name( *tag_it, tag_name );
3774  if( tag_name.c_str()[0] == '_' && tag_name.c_str()[1] == '_' ) continue;
3775 
3776  Range tmp_range;
3777  result = ( *tag_it )->get_tagged_entities( sequenceManager, tmp_range );
3778  MB_CHK_SET_ERR( result, "Failed to get entities for tag in pack_tags" );
3779  tmp_range = intersect( tmp_range, whole_range );
3780 
3781  if( tmp_range.empty() ) continue;
3782 
3783  // OK, we'll be sending this tag
3784  all_tags.push_back( *tag_it );
3785  tag_ranges.push_back( Range() );
3786  tag_ranges.back().swap( tmp_range );
3787  }
3788 
3789  return MB_SUCCESS;
3790 }

References moab::Range::empty(), ErrorCode, moab::intersect(), MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, sequenceManager, moab::Interface::tag_get_name(), and moab::Interface::tag_get_tags().

Referenced by pack_buffer().

◆ initialize()

void moab::ParallelComm::initialize ( )
private

Definition at line 341 of file ParallelComm.cpp.

342 {
343  Core* core = dynamic_cast< Core* >( mbImpl );
344  sequenceManager = core->sequence_manager();
345  mbImpl->query_interface( errorHandler );
346 
347  // Initialize MPI, if necessary
348  int flag = 1;
349  int retval = MPI_Initialized( &flag );
350  if( MPI_SUCCESS != retval || !flag )
351  {
352  int argc = 0;
353  char** argv = NULL;
354 
355  // mpi not initialized yet - initialize here
356  retval = MPI_Init( &argc, &argv );
357  assert( MPI_SUCCESS == retval );
358  }
359 
360  // Reserve space for vectors
361  buffProcs.reserve( MAX_SHARING_PROCS );
362  localOwnedBuffs.reserve( MAX_SHARING_PROCS );
363  remoteOwnedBuffs.reserve( MAX_SHARING_PROCS );
364 
365  pcommID = add_pcomm( this );
366 
367  if( !myDebug )
368  {
369  myDebug = new DebugOutput( "ParallelComm", std::cerr );
370  myDebug->set_rank( procConfig.proc_rank() );
371  }
372 }

References add_pcomm(), buffProcs, errorHandler, localOwnedBuffs, MAX_SHARING_PROCS, mbImpl, myDebug, pcommID, moab::ProcConfig::proc_rank(), procConfig, moab::Interface::query_interface(), remoteOwnedBuffs, moab::Core::sequence_manager(), sequenceManager, and moab::DebugOutput::set_rank().

Referenced by ParallelComm().

◆ interface_sets() [1/2]

Range& moab::ParallelComm::interface_sets ( )
inline

Definition at line 673 of file ParallelComm.hpp.

674  {
675  return interfaceSets;
676  }

References interfaceSets.

Referenced by check_clean_iface(), moab::NCHelperScrip::create_mesh(), and get_interface_sets().

◆ interface_sets() [2/2]

const Range& moab::ParallelComm::interface_sets ( ) const
inline

Definition at line 677 of file ParallelComm.hpp.

678  {
679  return interfaceSets;
680  }

References interfaceSets.

◆ is_iface_proc()

bool moab::ParallelComm::is_iface_proc ( EntityHandle  this_set,
int  to_proc 
)
private

returns true if the set is an interface shared with to_proc

Definition at line 5757 of file ParallelComm.cpp.

5758 {
5759  int sharing_procs[MAX_SHARING_PROCS];
5760  std::fill( sharing_procs, sharing_procs + MAX_SHARING_PROCS, -1 );
5761  ErrorCode result = mbImpl->tag_get_data( sharedp_tag(), &this_set, 1, sharing_procs );
5762  if( MB_SUCCESS == result && to_proc == sharing_procs[0] ) return true;
5763 
5764  result = mbImpl->tag_get_data( sharedps_tag(), &this_set, 1, sharing_procs );
5765  if( MB_SUCCESS != result ) return false;
5766 
5767  for( int i = 0; i < MAX_SHARING_PROCS; i++ )
5768  {
5769  if( to_proc == sharing_procs[i] )
5770  return true;
5771  else if( -1 == sharing_procs[i] )
5772  return false;
5773  }
5774 
5775  return false;
5776 }

References ErrorCode, MAX_SHARING_PROCS, MB_SUCCESS, mbImpl, sharedp_tag(), sharedps_tag(), and moab::Interface::tag_get_data().

Referenced by get_ghosted_entities(), and get_iface_entities().

◆ list_entities() [1/2]

ErrorCode moab::ParallelComm::list_entities ( const EntityHandle *  ents,
int  num_ents 
)

Definition at line 2641 of file ParallelComm.cpp.

2642 {
2643  if( NULL == ents )
2644  {
2645  Range shared_ents;
2646  std::copy( sharedEnts.begin(), sharedEnts.end(), range_inserter( shared_ents ) );
2647  shared_ents.print( "Shared entities:\n" );
2648  return MB_SUCCESS;
2649  }
2650 
2651  unsigned char pstat;
2652  EntityHandle tmp_handles[MAX_SHARING_PROCS];
2653  int tmp_procs[MAX_SHARING_PROCS];
2654  unsigned int num_ps;
2655  ErrorCode result;
2656 
2657  for( int i = 0; i < num_ents; i++ )
2658  {
2659  result = mbImpl->list_entities( ents + i, 1 );
2660  MB_CHK_ERR( result );
2661  double coords[3];
2662  result = mbImpl->get_coords( ents + i, 1, coords );
2663  std::cout << " coords: " << coords[0] << " " << coords[1] << " " << coords[2] << "\n";
2664 
2665  result = get_sharing_data( ents[i], tmp_procs, tmp_handles, pstat, num_ps );
2666  MB_CHK_SET_ERR( result, "Failed to get sharing data" );
2667 
2668  std::cout << "Pstatus: ";
2669  if( !num_ps )
2670  std::cout << "local " << std::endl;
2671  else
2672  {
2673  if( pstat & PSTATUS_NOT_OWNED ) std::cout << "NOT_OWNED; ";
2674  if( pstat & PSTATUS_SHARED ) std::cout << "SHARED; ";
2675  if( pstat & PSTATUS_MULTISHARED ) std::cout << "MULTISHARED; ";
2676  if( pstat & PSTATUS_INTERFACE ) std::cout << "INTERFACE; ";
2677  if( pstat & PSTATUS_GHOST ) std::cout << "GHOST; ";
2678  std::cout << std::endl;
2679  for( unsigned int j = 0; j < num_ps; j++ )
2680  {
2681  std::cout << " proc " << tmp_procs[j] << " id (handle) " << mbImpl->id_from_handle( tmp_handles[j] )
2682  << "(" << tmp_handles[j] << ")" << std::endl;
2683  }
2684  }
2685  std::cout << std::endl;
2686  }
2687 
2688  return MB_SUCCESS;
2689 }

References ErrorCode, moab::Interface::get_coords(), get_sharing_data(), moab::Interface::id_from_handle(), moab::Interface::list_entities(), MAX_SHARING_PROCS, MB_CHK_ERR, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, moab::Range::print(), PSTATUS_GHOST, PSTATUS_INTERFACE, PSTATUS_MULTISHARED, PSTATUS_NOT_OWNED, PSTATUS_SHARED, and sharedEnts.

Referenced by build_sharedhps_list(), check_local_shared(), check_my_shared_handles(), list_entities(), moab::ReadParallel::load_file(), and moab::ScdInterface::tag_shared_vertices().

◆ list_entities() [2/2]

ErrorCode moab::ParallelComm::list_entities ( const Range &  ents)

Definition at line 2691 of file ParallelComm.cpp.

2692 {
2693  for( Range::iterator rit = ents.begin(); rit != ents.end(); ++rit )
2694  list_entities( &( *rit ), 1 );
2695 
2696  return MB_SUCCESS;
2697 }

References moab::Range::begin(), moab::Range::end(), list_entities(), and MB_SUCCESS.

◆ pack_adjacencies()

ErrorCode moab::ParallelComm::pack_adjacencies ( Range &  entities,
Range::const_iterator &  start_rit,
Range &  whole_range,
unsigned char *&  buff_ptr,
int &  count,
const bool  just_count,
const bool  store_handles,
const int  to_proc 
)
private

Definition at line 3557 of file ParallelComm.cpp.

3565 {
3566  return MB_FAILURE;
3567 }

◆ pack_buffer()

ErrorCode moab::ParallelComm::pack_buffer ( Range &  orig_ents,
const bool  adjacencies,
const bool  tags,
const bool  store_remote_handles,
const int  to_proc,
Buffer *  buff,
TupleList *  entprocs = NULL,
Range *  allsent = NULL 
)

public 'cuz we want to unit test these externally

Definition at line 1448 of file ParallelComm.cpp.

1456 {
1457  // Pack the buffer with the entity ranges, adjacencies, and tags sections
1458  //
1459  // Note: new entities used in subsequent connectivity lists, sets, or tags,
1460  // are referred to as (MBMAXTYPE + index), where index is into vector
1461  // of new entities, 0-based
1462  ErrorCode result;
1463 
1464  Range set_range;
1465  std::vector< Tag > all_tags;
1466  std::vector< Range > tag_ranges;
1467 
1468  Range::const_iterator rit;
1469 
1470  // Entities
1471  result = pack_entities( orig_ents, buff, store_remote_handles, to_proc, false, entprocs, allsent );
1472  MB_CHK_SET_ERR( result, "Packing entities failed" );
1473 
1474  // Sets
1475  result = pack_sets( orig_ents, buff, store_remote_handles, to_proc );
1476  MB_CHK_SET_ERR( result, "Packing sets (count) failed" );
1477 
1478  // Tags
1479  Range final_ents;
1480  if( tags )
1481  {
1482  result = get_tag_send_list( orig_ents, all_tags, tag_ranges );
1483  MB_CHK_SET_ERR( result, "Failed to get tagged entities" );
1484  result = pack_tags( orig_ents, all_tags, all_tags, tag_ranges, buff, store_remote_handles, to_proc );
1485  MB_CHK_SET_ERR( result, "Packing tags (count) failed" );
1486  }
1487  else
1488  { // Set tag size to 0
1489  buff->check_space( sizeof( int ) );
1490  PACK_INT( buff->buff_ptr, 0 );
1491  buff->set_stored_size();
1492  }
1493 
1494  return result;
1495 }

References moab::ParallelComm::Buffer::buff_ptr, moab::ParallelComm::Buffer::check_space(), ErrorCode, get_tag_send_list(), MB_CHK_SET_ERR, pack_entities(), moab::PACK_INT(), pack_sets(), pack_tags(), and moab::ParallelComm::Buffer::set_stored_size().

Referenced by broadcast_entities(), exchange_owned_mesh(), scatter_entities(), send_entities(), and moab::ParCommGraph::send_mesh_parts().

◆ pack_entities()

ErrorCode moab::ParallelComm::pack_entities ( Range &  entities,
Buffer *  buff,
const bool  store_remote_handles,
const int  to_proc,
const bool  is_iface,
TupleList *  entprocs = NULL,
Range *  allsent = NULL 
)

Definition at line 1625 of file ParallelComm.cpp.

1632 {
1633  // Packed information:
1634  // 1. # entities = E
1635  // 2. for e in E
1636  // a. # procs sharing e, incl. sender and receiver = P
1637  // b. for p in P (procs sharing e)
1638  // c. for p in P (handle for e on p) (Note1)
1639  // 3. vertex/entity info
1640 
1641  // Get an estimate of the buffer size & pre-allocate buffer size
1642  int buff_size = estimate_ents_buffer_size( entities, store_remote_handles );
1643  if( buff_size < 0 ) MB_SET_ERR( MB_FAILURE, "Failed to estimate ents buffer size" );
1644  buff->check_space( buff_size );
1645  myDebug->tprintf( 3, "estimate buffer size for %d entities: %d \n", (int)entities.size(), buff_size );
1646 
1647  unsigned int num_ents;
1648  ErrorCode result;
1649 
1650  std::vector< EntityHandle > entities_vec( entities.size() );
1651  std::copy( entities.begin(), entities.end(), entities_vec.begin() );
1652 
1653  // First pack procs/handles sharing this ent, not including this dest but including
1654  // others (with zero handles)
1655  if( store_remote_handles )
1656  {
1657  // Buff space is at least proc + handle for each entity; use avg of 4 other procs
1658  // to estimate buff size, but check later
1659  buff->check_space( sizeof( int ) + ( 5 * sizeof( int ) + sizeof( EntityHandle ) ) * entities.size() );
1660 
1661  // 1. # entities = E
1662  PACK_INT( buff->buff_ptr, entities.size() );
1663 
1664  Range::iterator rit;
1665 
1666  // Pre-fetch sharedp and pstatus
1667  std::vector< int > sharedp_vals( entities.size() );
1668  result = mbImpl->tag_get_data( sharedp_tag(), entities, sharedp_vals.data() );
1669  MB_CHK_SET_ERR( result, "Failed to get sharedp tag data" );
1670  std::vector< char > pstatus_vals( entities.size() );
1671  result = mbImpl->tag_get_data( pstatus_tag(), entities, pstatus_vals.data() );
1672  MB_CHK_SET_ERR( result, "Failed to get pstatus tag data" );
1673 
1674  unsigned int i;
1675  int tmp_procs[MAX_SHARING_PROCS];
1676  EntityHandle tmp_handles[MAX_SHARING_PROCS];
1677  std::set< unsigned int > dumprocs;
1678 
1679  // 2. for e in E
1680  for( rit = entities.begin(), i = 0; rit != entities.end(); ++rit, i++ )
1681  {
1682  unsigned int ind =
1683  std::lower_bound( entprocs->vul_rd, entprocs->vul_rd + entprocs->get_n(), *rit ) - entprocs->vul_rd;
1684  assert( ind < entprocs->get_n() );
1685 
1686  while( ind < entprocs->get_n() && entprocs->vul_rd[ind] == *rit )
1687  dumprocs.insert( entprocs->vi_rd[ind++] );
1688 
1689  result = build_sharedhps_list( *rit, pstatus_vals[i], sharedp_vals[i], dumprocs, num_ents, tmp_procs,
1690  tmp_handles );
1691  MB_CHK_SET_ERR( result, "Failed to build sharedhps" );
1692 
1693  dumprocs.clear();
1694 
1695  // Now pack them
1696  buff->check_space( ( num_ents + 1 ) * sizeof( int ) + num_ents * sizeof( EntityHandle ) );
1697  PACK_INT( buff->buff_ptr, num_ents );
1698  PACK_INTS( buff->buff_ptr, tmp_procs, num_ents );
1699  PACK_EH( buff->buff_ptr, tmp_handles, num_ents );
1700 
1701 #ifndef NDEBUG
1702  // Check for duplicates in proc list
1703  unsigned int dp = 0;
1704  for( ; dp < MAX_SHARING_PROCS && -1 != tmp_procs[dp]; dp++ )
1705  dumprocs.insert( tmp_procs[dp] );
1706  assert( dumprocs.size() == dp );
1707  dumprocs.clear();
1708 #endif
1709  }
1710  }
1711 
1712  // Pack vertices
1713  Range these_ents = entities.subset_by_type( MBVERTEX );
1714  num_ents = these_ents.size();
1715 
1716  if( num_ents )
1717  {
1718  buff_size = 2 * sizeof( int ) + 3 * num_ents * sizeof( double );
1719  buff->check_space( buff_size );
1720 
1721  // Type, # ents
1722  PACK_INT( buff->buff_ptr, ( (int)MBVERTEX ) );
1723  PACK_INT( buff->buff_ptr, ( (int)num_ents ) );
1724 
1725  std::vector< double > tmp_coords( 3 * num_ents );
1726  result = mbImpl->get_coords( these_ents, tmp_coords.data() );
1727  MB_CHK_SET_ERR( result, "Failed to get vertex coordinates" );
1728  PACK_DBLS( buff->buff_ptr, tmp_coords.data(), 3 * num_ents );
1729 
1730  myDebug->tprintf( 4, "Packed %lu ents of type %s\n", (unsigned long)these_ents.size(),
1731  CN::EntityTypeName( TYPE_FROM_HANDLE( *these_ents.begin() ) ) );
1732  }
1733 
1734  // Now entities; go through range, packing by type and equal # verts per element
1735  Range::iterator start_rit = entities.find( *these_ents.rbegin() );
1736  ++start_rit;
1737  int last_nodes = -1;
1738  EntityType last_type = MBMAXTYPE;
1739  these_ents.clear();
1740  Range::iterator end_rit = start_rit;
1741  EntitySequence* seq;
1742  ElementSequence* eseq;
1743 
1744  while( start_rit != entities.end() || !these_ents.empty() )
1745  {
1746  // Cases:
1747  // A: !end, last_type == MBMAXTYPE, seq: save contig sequence in these_ents
1748  // B: !end, last type & nodes same, seq: save contig sequence in these_ents
1749  // C: !end, last type & nodes different: pack these_ents, then save contig sequence in
1750  // these_ents D: end: pack these_ents
1751 
1752  // Find the sequence holding current start entity, if we're not at end
1753  eseq = NULL;
1754  if( start_rit != entities.end() )
1755  {
1756  result = sequenceManager->find( *start_rit, seq );
1757  MB_CHK_SET_ERR( result, "Failed to find entity sequence" );
1758  if( NULL == seq ) return MB_FAILURE;
1759  eseq = dynamic_cast< ElementSequence* >( seq );
1760  }
1761 
1762  // Pack the last batch if at end or next one is different
1763  if( !these_ents.empty() &&
1764  ( !eseq || eseq->type() != last_type || last_nodes != (int)eseq->nodes_per_element() ) )
1765  {
1766  result = pack_entity_seq( last_nodes, store_remote_handles, to_proc, these_ents, entities_vec, buff );
1767  MB_CHK_SET_ERR( result, "Failed to pack entities from a sequence" );
1768  these_ents.clear();
1769  }
1770 
1771  if( eseq )
1772  {
1773  // Continuation of current range, just save these entities
1774  // Get position in entities list one past end of this sequence
1775  end_rit = entities.lower_bound( start_rit, entities.end(), eseq->end_handle() + 1 );
1776 
1777  // Put these entities in the range
1778  std::copy( start_rit, end_rit, range_inserter( these_ents ) );
1779 
1780  last_type = eseq->type();
1781  last_nodes = eseq->nodes_per_element();
1782  }
1783  else if( start_rit != entities.end() && TYPE_FROM_HANDLE( *start_rit ) == MBENTITYSET )
1784  break;
1785 
1786  start_rit = end_rit;
1787  }
1788 
1789  // Pack MBMAXTYPE to indicate end of ranges
1790  buff->check_space( sizeof( int ) );
1791  PACK_INT( buff->buff_ptr, ( (int)MBMAXTYPE ) );
1792 
1793  buff->set_stored_size();
1794  return MB_SUCCESS;
1795 }

References moab::Range::begin(), moab::ParallelComm::Buffer::buff_ptr, build_sharedhps_list(), moab::ParallelComm::Buffer::check_space(), moab::Range::clear(), moab::Range::empty(), moab::Range::end(), moab::EntitySequence::end_handle(), moab::CN::EntityTypeName(), ErrorCode, estimate_ents_buffer_size(), moab::SequenceManager::find(), moab::Range::find(), moab::Interface::get_coords(), moab::TupleList::get_n(), moab::Range::lower_bound(), MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SET_ERR, MB_SUCCESS, MBENTITYSET, mbImpl, MBMAXTYPE, MBVERTEX, myDebug, moab::ElementSequence::nodes_per_element(), moab::PACK_DBLS(), moab::PACK_EH(), pack_entity_seq(), moab::PACK_INT(), moab::PACK_INTS(), pstatus_tag(), moab::Range::rbegin(), sequenceManager, moab::ParallelComm::Buffer::set_stored_size(), sharedp_tag(), moab::Range::size(), moab::Range::subset_by_type(), moab::Interface::tag_get_data(), moab::DebugOutput::tprintf(), moab::EntitySequence::type(), moab::TYPE_FROM_HANDLE(), moab::TupleList::vi_rd, and moab::TupleList::vul_rd.

Referenced by exchange_ghost_cells(), and pack_buffer().

◆ pack_entity_seq()

ErrorCode moab::ParallelComm::pack_entity_seq ( const int  nodes_per_entity,
const bool  store_remote_handles,
const int  to_proc,
Range &  these_ents,
std::vector< EntityHandle > &  entities,
Buffer *  buff 
)
private

pack a range of entities with equal # verts per entity, along with the range on the sending proc

Definition at line 1886 of file ParallelComm.cpp.

1892 {
1893  int tmp_space = 3 * sizeof( int ) + nodes_per_entity * these_ents.size() * sizeof( EntityHandle );
1894  buff->check_space( tmp_space );
1895 
1896  // Pack the entity type
1897  PACK_INT( buff->buff_ptr, ( (int)TYPE_FROM_HANDLE( *these_ents.begin() ) ) );
1898 
1899  // Pack # ents
1900  PACK_INT( buff->buff_ptr, these_ents.size() );
1901 
1902  // Pack the nodes per entity
1903  PACK_INT( buff->buff_ptr, nodes_per_entity );
1904  myDebug->tprintf( 3, "after some pack int %d \n", buff->get_current_size() );
1905 
1906  // Pack the connectivity
1907  std::vector< EntityHandle > connect;
1908  ErrorCode result = MB_SUCCESS;
1909  for( Range::const_iterator rit = these_ents.begin(); rit != these_ents.end(); ++rit )
1910  {
1911  connect.clear();
1912  result = mbImpl->get_connectivity( &( *rit ), 1, connect, false );
1913  MB_CHK_SET_ERR( result, "Failed to get connectivity" );
1914  assert( (int)connect.size() == nodes_per_entity );
1915  result = get_remote_handles( store_remote_handles, connect.data(), connect.data(), connect.size(), to_proc,
1916  entities_vec );
1917  MB_CHK_SET_ERR( result, "Failed in get_remote_handles" );
1918  PACK_EH( buff->buff_ptr, connect.data(), connect.size() );
1919  }
1920 
1921  myDebug->tprintf( 3, "Packed %lu ents of type %s\n", (unsigned long)these_ents.size(),
1922  CN::EntityTypeName( TYPE_FROM_HANDLE( *these_ents.begin() ) ) );
1923 
1924  return result;
1925 }

References moab::Range::begin(), moab::ParallelComm::Buffer::buff_ptr, moab::ParallelComm::Buffer::check_space(), moab::Range::end(), moab::CN::EntityTypeName(), ErrorCode, moab::Interface::get_connectivity(), moab::ParallelComm::Buffer::get_current_size(), get_remote_handles(), MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, myDebug, moab::PACK_EH(), moab::PACK_INT(), moab::Range::size(), moab::DebugOutput::tprintf(), and moab::TYPE_FROM_HANDLE().

Referenced by pack_entities().

◆ pack_range_map()

ErrorCode moab::ParallelComm::pack_range_map ( Range &  this_range,
EntityHandle  actual_start,
HandleMap &  handle_map 
)
private

pack a range map with keys in this_range and values a contiguous series of handles starting at actual_start

Definition at line 3224 of file ParallelComm.cpp.

3225 {
3226  for( Range::const_pair_iterator key_it = key_range.const_pair_begin(); key_it != key_range.const_pair_end();
3227  ++key_it )
3228  {
3229  int tmp_num = ( *key_it ).second - ( *key_it ).first + 1;
3230  handle_map.insert( ( *key_it ).first, val_start, tmp_num );
3231  val_start += tmp_num;
3232  }
3233 
3234  return MB_SUCCESS;
3235 }

References moab::Range::const_pair_begin(), moab::Range::const_pair_end(), moab::RangeMap< KeyType, ValType, NullVal >::insert(), and MB_SUCCESS.

◆ pack_remote_handles()

ErrorCode moab::ParallelComm::pack_remote_handles ( std::vector< EntityHandle > &  L1hloc,
std::vector< EntityHandle > &  L1hrem,
std::vector< int > &  procs,
unsigned int  to_proc,
Buffer *  buff 
)

Definition at line 7651 of file ParallelComm.cpp.

7656 {
7657  assert( std::find( L1hloc.begin(), L1hloc.end(), (EntityHandle)0 ) == L1hloc.end() );
7658 
7659  // 2 vectors of handles plus ints
7660  buff->check_space( ( ( L1p.size() + 1 ) * sizeof( int ) + ( L1hloc.size() + 1 ) * sizeof( EntityHandle ) +
7661  ( L1hrem.size() + 1 ) * sizeof( EntityHandle ) ) );
7662 
7663  // Should be in pairs of handles
7664  PACK_INT( buff->buff_ptr, L1hloc.size() );
7665  PACK_INTS( buff->buff_ptr, L1p.data(), L1p.size() );
7666  // Pack handles in reverse order, (remote, local), so on destination they
7667  // are ordered (local, remote)
7668  PACK_EH( buff->buff_ptr, L1hrem.data(), L1hrem.size() );
7669  PACK_EH( buff->buff_ptr, L1hloc.data(), L1hloc.size() );
7670 
7671  buff->set_stored_size();
7672 
7673  return MB_SUCCESS;
7674 }

References moab::ParallelComm::Buffer::buff_ptr, moab::ParallelComm::Buffer::check_space(), MB_SUCCESS, moab::PACK_EH(), moab::PACK_INT(), moab::PACK_INTS(), and moab::ParallelComm::Buffer::set_stored_size().

Referenced by exchange_ghost_cells(), exchange_owned_mesh(), recv_entities(), and recv_messages().

◆ pack_sets()

ErrorCode moab::ParallelComm::pack_sets ( Range &  entities,
Buffer *  buff,
const bool  store_handles,
const int  to_proc 
)
private

Definition at line 3237 of file ParallelComm.cpp.

3238 {
3239  // SETS:
3240  // . #sets
3241  // . for each set:
3242  // - options[#sets] (unsigned int)
3243  // - if (unordered) set range
3244  // - else if ordered
3245  // . #ents in set
3246  // . handles[#ents]
3247  // - #parents
3248  // - if (#parents) handles[#parents]
3249  // - #children
3250  // - if (#children) handles[#children]
3251 
3252  // Now the sets; assume any sets the application wants to pass are in the entities list
3253  ErrorCode result;
3254  Range all_sets = entities.subset_by_type( MBENTITYSET );
3255 
3256  int buff_size = estimate_sets_buffer_size( all_sets, store_remote_handles );
3257  if( buff_size < 0 ) MB_SET_ERR( MB_FAILURE, "Failed to estimate sets buffer size" );
3258  buff->check_space( buff_size );
3259 
3260  // Number of sets
3261  PACK_INT( buff->buff_ptr, all_sets.size() );
3262 
3263  // Options for all sets
3264  std::vector< unsigned int > options( all_sets.size() );
3265  Range::iterator rit;
3266  std::vector< EntityHandle > members;
3267  int i;
3268  for( rit = all_sets.begin(), i = 0; rit != all_sets.end(); ++rit, i++ )
3269  {
3270  result = mbImpl->get_meshset_options( *rit, options[i] );
3271  MB_CHK_SET_ERR( result, "Failed to get meshset options" );
3272  }
3273  buff->check_space( all_sets.size() * sizeof( unsigned int ) );
3274  PACK_VOID( buff->buff_ptr, options.data(), all_sets.size() * sizeof( unsigned int ) );
3275 
3276  // Pack parallel geometry unique id
3277  if( !all_sets.empty() )
3278  {
3279  Tag uid_tag;
3280  int n_sets = all_sets.size();
3281  bool b_pack = false;
3282  std::vector< int > id_data( n_sets );
3283  result =
3284  mbImpl->tag_get_handle( "PARALLEL_UNIQUE_ID", 1, MB_TYPE_INTEGER, uid_tag, MB_TAG_SPARSE | MB_TAG_CREAT );
3285  MB_CHK_SET_ERR( result, "Failed to create parallel geometry unique id tag" );
3286 
3287  result = mbImpl->tag_get_data( uid_tag, all_sets, id_data.data() );
3288  if( MB_TAG_NOT_FOUND != result )
3289  {
3290  if( MB_SUCCESS != result ) MB_SET_ERR( result, "Failed to get parallel geometry unique ids" );
3291  for( i = 0; i < n_sets; i++ )
3292  {
3293  if( id_data[i] != 0 )
3294  {
3295  b_pack = true;
3296  break;
3297  }
3298  }
3299  }
3300 
3301  if( b_pack )
3302  { // If you find
3303  buff->check_space( ( n_sets + 1 ) * sizeof( int ) );
3304  PACK_INT( buff->buff_ptr, n_sets );
3305  PACK_INTS( buff->buff_ptr, id_data.data(), n_sets );
3306  }
3307  else
3308  {
3309  buff->check_space( sizeof( int ) );
3310  PACK_INT( buff->buff_ptr, 0 );
3311  }
3312  }
3313 
3314  // Vectors/ranges
3315  std::vector< EntityHandle > entities_vec( entities.size() );
3316  std::copy( entities.begin(), entities.end(), entities_vec.begin() );
3317  for( rit = all_sets.begin(), i = 0; rit != all_sets.end(); ++rit, i++ )
3318  {
3319  members.clear();
3320  result = mbImpl->get_entities_by_handle( *rit, members );
3321  MB_CHK_SET_ERR( result, "Failed to get entities in ordered set" );
3322  result = get_remote_handles( store_remote_handles, members.data(), members.data(), members.size(), to_proc,
3323  entities_vec );
3324  MB_CHK_SET_ERR( result, "Failed in get_remote_handles" );
3325  buff->check_space( members.size() * sizeof( EntityHandle ) + sizeof( int ) );
3326  PACK_INT( buff->buff_ptr, members.size() );
3327  PACK_EH( buff->buff_ptr, members.data(), members.size() );
3328  }
3329 
3330  // Pack parent/child sets
3331  if( !store_remote_handles )
3332  { // Only works not store remote handles
3333  // Pack numbers of parents/children
3334  unsigned int tot_pch = 0;
3335  int num_pch;
3336  buff->check_space( 2 * all_sets.size() * sizeof( int ) );
3337  for( rit = all_sets.begin(), i = 0; rit != all_sets.end(); ++rit, i++ )
3338  {
3339  // Pack parents
3340  result = mbImpl->num_parent_meshsets( *rit, &num_pch );
3341  MB_CHK_SET_ERR( result, "Failed to get num parents" );
3342  PACK_INT( buff->buff_ptr, num_pch );
3343  tot_pch += num_pch;
3344  result = mbImpl->num_child_meshsets( *rit, &num_pch );
3345  MB_CHK_SET_ERR( result, "Failed to get num children" );
3346  PACK_INT( buff->buff_ptr, num_pch );
3347  tot_pch += num_pch;
3348  }
3349 
3350  // Now pack actual parents/children
3351  members.clear();
3352  members.reserve( tot_pch );
3353  std::vector< EntityHandle > tmp_pch;
3354  for( rit = all_sets.begin(), i = 0; rit != all_sets.end(); ++rit, i++ )
3355  {
3356  result = mbImpl->get_parent_meshsets( *rit, tmp_pch );
3357  MB_CHK_SET_ERR( result, "Failed to get parents" );
3358  std::copy( tmp_pch.begin(), tmp_pch.end(), std::back_inserter( members ) );
3359  tmp_pch.clear();
3360  result = mbImpl->get_child_meshsets( *rit, tmp_pch );
3361  MB_CHK_SET_ERR( result, "Failed to get children" );
3362  std::copy( tmp_pch.begin(), tmp_pch.end(), std::back_inserter( members ) );
3363  tmp_pch.clear();
3364  }
3365  assert( members.size() == tot_pch );
3366  if( !members.empty() )
3367  {
3368  result = get_remote_handles( store_remote_handles, members.data(), members.data(), members.size(), to_proc,
3369  entities_vec );
3370  MB_CHK_SET_ERR( result, "Failed to get remote handles for set parent/child sets" );
3371 #ifndef NDEBUG
3372  // Check that all handles are either sets or maxtype
3373  for( unsigned int __j = 0; __j < members.size(); __j++ )
3374  assert( ( TYPE_FROM_HANDLE( members[__j] ) == MBMAXTYPE &&
3375  ID_FROM_HANDLE( members[__j] ) < (int)entities.size() ) ||
3376  TYPE_FROM_HANDLE( members[__j] ) == MBENTITYSET );
3377 #endif
3378  buff->check_space( members.size() * sizeof( EntityHandle ) );
3379  PACK_EH( buff->buff_ptr, members.data(), members.size() );
3380  }
3381  }
3382  else
3383  {
3384  buff->check_space( 2 * all_sets.size() * sizeof( int ) );
3385  for( rit = all_sets.begin(); rit != all_sets.end(); ++rit )
3386  {
3387  PACK_INT( buff->buff_ptr, 0 );
3388  PACK_INT( buff->buff_ptr, 0 );
3389  }
3390  }
3391 
3392  // Pack the handles
3393  if( store_remote_handles && !all_sets.empty() )
3394  {
3395  buff_size = RANGE_SIZE( all_sets );
3396  buff->check_space( buff_size );
3397  PACK_RANGE( buff->buff_ptr, all_sets );
3398  }
3399 
3400  myDebug->tprintf( 4, "Done packing sets.\n" );
3401 
3402  buff->set_stored_size();
3403 
3404  return MB_SUCCESS;
3405 }

References moab::Range::begin(), moab::ParallelComm::Buffer::buff_ptr, moab::ParallelComm::Buffer::check_space(), moab::Range::empty(), moab::Range::end(), ErrorCode, estimate_sets_buffer_size(), moab::Interface::get_child_meshsets(), moab::Interface::get_entities_by_handle(), moab::Interface::get_meshset_options(), moab::Interface::get_parent_meshsets(), get_remote_handles(), moab::ID_FROM_HANDLE(), MB_CHK_SET_ERR, MB_SET_ERR, MB_SUCCESS, MB_TAG_CREAT, MB_TAG_NOT_FOUND, MB_TAG_SPARSE, MB_TYPE_INTEGER, MBENTITYSET, mbImpl, MBMAXTYPE, myDebug, moab::Interface::num_child_meshsets(), moab::Interface::num_parent_meshsets(), moab::PACK_EH(), moab::PACK_INT(), moab::PACK_INTS(), moab::PACK_RANGE(), moab::PACK_VOID(), moab::RANGE_SIZE(), moab::ParallelComm::Buffer::set_stored_size(), moab::Range::size(), moab::Range::subset_by_type(), moab::Interface::tag_get_data(), moab::Interface::tag_get_handle(), moab::DebugOutput::tprintf(), and moab::TYPE_FROM_HANDLE().

Referenced by pack_buffer().

◆ pack_shared_handles()

ErrorCode moab::ParallelComm::pack_shared_handles ( std::vector< std::vector< SharedEntityData > > &  send_data)

Definition at line 8769 of file ParallelComm.cpp.

8770 {
8771  // Build up send buffers
8772  ErrorCode rval = MB_SUCCESS;
8773  int ent_procs[MAX_SHARING_PROCS];
8774  EntityHandle handles[MAX_SHARING_PROCS];
8775  int num_sharing, tmp_int;
8776  SharedEntityData tmp;
8777  send_data.resize( buffProcs.size() );
8778  for( std::set< EntityHandle >::iterator i = sharedEnts.begin(); i != sharedEnts.end(); ++i )
8779  {
8780  tmp.remote = *i; // Swap local/remote so they're correct on the remote proc.
8781  rval = get_owner( *i, tmp_int );
8782  tmp.owner = tmp_int;
8783  if( MB_SUCCESS != rval ) return rval;
8784 
8785  unsigned char pstat;
8786  rval = get_sharing_data( *i, ent_procs, handles, pstat, num_sharing );
8787  if( MB_SUCCESS != rval ) return rval;
8788  for( int j = 0; j < num_sharing; j++ )
8789  {
8790  if( ent_procs[j] == (int)proc_config().proc_rank() ) continue;
8791  tmp.local = handles[j];
8792  int ind = get_buffers( ent_procs[j] );
8793  assert( -1 != ind );
8794  if( (int)send_data.size() < ind + 1 ) send_data.resize( ind + 1 );
8795  send_data[ind].push_back( tmp );
8796  }
8797  }
8798 
8799  return MB_SUCCESS;
8800 }

References buffProcs, ErrorCode, get_buffers(), get_owner(), get_sharing_data(), moab::ParallelComm::SharedEntityData::local, MAX_SHARING_PROCS, MB_SUCCESS, moab::ParallelComm::SharedEntityData::owner, proc_config(), moab::ParallelComm::SharedEntityData::remote, and sharedEnts.

Referenced by check_all_shared_handles().

◆ pack_tag()

ErrorCode moab::ParallelComm::pack_tag ( Tag  source_tag,
Tag  destination_tag,
const Range &  entities,
const std::vector< EntityHandle > &  whole_range,
Buffer *  buff,
const bool  store_remote_handles,
const int  to_proc 
)
private

Serialize tag data.

Parameters
source_tagThe tag for which data will be serialized
destination_tagTag in which to store unpacked tag data. Typically the same as source_tag.
entitiesThe entities for which tag values will be serialized
whole_rangeCalculate entity indices as location in this range
buff_ptrInput/Output: As input, pointer to the start of the buffer in which to serialize data. As output, the position just passed the serialized data.
count_outOutput: The required buffer size, in bytes.
store_handlesThe data for each tag is preceded by a list of EntityHandles designating the entity each of the subsequent tag values corresponds to. This value may be one of: 1) If store_handles == false: An invalid handle composed of {MBMAXTYPE,idx}, where idx is the position of the entity in "whole_range". 2) If store_hanldes == true and a valid remote handle exists, the remote handle. 3) If store_hanldes == true and no valid remote handle is defined for the entity, the same as 1).
to_procIf 'store_handles' is true, the processor rank for which to store the corresponding remote entity handles.

Definition at line 3664 of file ParallelComm.cpp.

3671 {
3672  ErrorCode result;
3673  std::vector< int > var_len_sizes;
3674  std::vector< const void* > var_len_values;
3675 
3676  if( src_tag != dst_tag )
3677  {
3678  if( dst_tag->get_size() != src_tag->get_size() ) return MB_TYPE_OUT_OF_RANGE;
3679  if( dst_tag->get_data_type() != src_tag->get_data_type() && dst_tag->get_data_type() != MB_TYPE_OPAQUE &&
3680  src_tag->get_data_type() != MB_TYPE_OPAQUE )
3681  return MB_TYPE_OUT_OF_RANGE;
3682  }
3683 
3684  // Size, type, data type
3685  buff->check_space( 3 * sizeof( int ) );
3686  PACK_INT( buff->buff_ptr, src_tag->get_size() );
3687  TagType this_type;
3688  result = mbImpl->tag_get_type( dst_tag, this_type );
3689  PACK_INT( buff->buff_ptr, (int)this_type );
3690  DataType data_type = src_tag->get_data_type();
3691  PACK_INT( buff->buff_ptr, (int)data_type );
3692  int type_size = TagInfo::size_from_data_type( data_type );
3693 
3694  // Default value
3695  if( NULL == src_tag->get_default_value() )
3696  {
3697  buff->check_space( sizeof( int ) );
3698  PACK_INT( buff->buff_ptr, 0 );
3699  }
3700  else
3701  {
3702  buff->check_space( src_tag->get_default_value_size() );
3703  PACK_BYTES( buff->buff_ptr, src_tag->get_default_value(), src_tag->get_default_value_size() );
3704  }
3705 
3706  // Name
3707  buff->check_space( src_tag->get_name().size() );
3708  PACK_BYTES( buff->buff_ptr, dst_tag->get_name().c_str(), dst_tag->get_name().size() );
3709 
3710  myDebug->tprintf( 4, "Packing tag \"%s\"", src_tag->get_name().c_str() );
3711  if( src_tag != dst_tag ) myDebug->tprintf( 4, " (as tag \"%s\")", dst_tag->get_name().c_str() );
3712  myDebug->tprintf( 4, "\n" );
3713 
3714  // Pack entities
3715  buff->check_space( tagged_entities.size() * sizeof( EntityHandle ) + sizeof( int ) );
3716  PACK_INT( buff->buff_ptr, tagged_entities.size() );
3717  std::vector< EntityHandle > dum_tagged_entities( tagged_entities.size() );
3718  result =
3719  get_remote_handles( store_remote_handles, tagged_entities, dum_tagged_entities.data(), to_proc, whole_vec );
3720  if( MB_SUCCESS != result )
3721  {
3722  if( myDebug->get_verbosity() == 3 )
3723  {
3724  std::cerr << "Failed to get remote handles for tagged entities:" << std::endl;
3725  tagged_entities.print( " " );
3726  }
3727  MB_SET_ERR( result, "Failed to get remote handles for tagged entities" );
3728  }
3729 
3730  PACK_EH( buff->buff_ptr, dum_tagged_entities.data(), dum_tagged_entities.size() );
3731 
3732  const size_t num_ent = tagged_entities.size();
3733  if( src_tag->get_size() == MB_VARIABLE_LENGTH )
3734  {
3735  var_len_sizes.resize( num_ent, 0 );
3736  var_len_values.resize( num_ent, 0 );
3737  result = mbImpl->tag_get_by_ptr( src_tag, tagged_entities, var_len_values.data(), var_len_sizes.data() );
3738  MB_CHK_SET_ERR( result, "Failed to get variable-length tag data in pack_tags" );
3739  buff->check_space( num_ent * sizeof( int ) );
3740  PACK_INTS( buff->buff_ptr, var_len_sizes.data(), num_ent );
3741  for( unsigned int i = 0; i < num_ent; i++ )
3742  {
3743  buff->check_space( var_len_sizes[i] );
3744  PACK_VOID( buff->buff_ptr, var_len_values[i], type_size * var_len_sizes[i] );
3745  }
3746  }
3747  else
3748  {
3749  buff->check_space( num_ent * src_tag->get_size() );
3750  // Should be OK to read directly into buffer, since tags are untyped and
3751  // handled by memcpy
3752  result = mbImpl->tag_get_data( src_tag, tagged_entities, buff->buff_ptr );
3753  MB_CHK_SET_ERR( result, "Failed to get tag data in pack_tags" );
3754  buff->buff_ptr += num_ent * src_tag->get_size();
3755  PC( num_ent * src_tag->get_size(), " void" );
3756  }
3757 
3758  return MB_SUCCESS;
3759 }

References moab::ParallelComm::Buffer::buff_ptr, moab::ParallelComm::Buffer::check_space(), ErrorCode, moab::TagInfo::get_data_type(), moab::TagInfo::get_default_value(), moab::TagInfo::get_default_value_size(), moab::TagInfo::get_name(), get_remote_handles(), moab::TagInfo::get_size(), moab::DebugOutput::get_verbosity(), MB_CHK_SET_ERR, MB_SET_ERR, MB_SUCCESS, MB_TYPE_OPAQUE, MB_TYPE_OUT_OF_RANGE, MB_VARIABLE_LENGTH, mbImpl, myDebug, moab::PACK_BYTES(), moab::PACK_EH(), moab::PACK_INT(), moab::PACK_INTS(), moab::PACK_VOID(), PC, moab::Range::print(), moab::Range::size(), moab::TagInfo::size_from_data_type(), moab::Interface::tag_get_by_ptr(), moab::Interface::tag_get_data(), moab::Interface::tag_get_type(), TagType, and moab::DebugOutput::tprintf().

Referenced by pack_tags().

◆ pack_tags()

ErrorCode moab::ParallelComm::pack_tags ( Range &  entities,
const std::vector< Tag > &  src_tags,
const std::vector< Tag > &  dst_tags,
const std::vector< Range > &  tag_ranges,
Buffer *  buff,
const bool  store_handles,
const int  to_proc 
)
private

Serialize entity tag data.

This function operates in two passes. The first phase, specified by 'just_count == true' calculates the necessary buffer size for the serialized data. The second phase writes the actual binary serialized representation of the data to the passed buffer.

Note
First two arguments are not used. (Legacy interface?)
Parameters
entitiesNOT USED
start_ritNOT USED
whole_rangeShould be the union of the sets of entities for which tag values are to be serialized. Also specifies ordering for indexes for tag values and serves as the superset from which to compose entity lists from individual tags if just_count and all_possible_tags are both true.
buff_ptrBuffer into which to write binary serialized data
countOutput: The size of the serialized data is added to this parameter. NOTE: Should probably initialize to zero before calling.
just_countIf true, just calculate the buffer size required to hold the serialized data. Will also append to 'all_tags' and 'tag_ranges' if all_possible_tags == true.
store_handlesThe data for each tag is preceded by a list of EntityHandles designating the entity each of the subsequent tag values corresponds to. This value may be one of: 1) If store_handles == false: An invalid handle composed of {MBMAXTYPE,idx}, where idx is the position of the entity in "whole_range". 2) If store_hanldes == true and a valid remote handle exists, the remote handle. 3) If store_hanldes == true and no valid remote handle is defined for the entity, the same as 1).
to_procIf 'store_handles' is true, the processor rank for which to store the corresponding remote entity handles.
all_tagsList of tags to write
tag_rangesList of entities to serialize tag data, one for each corresponding tag handle in 'all_tags.

Definition at line 3577 of file ParallelComm.cpp.

3584 {
3585  ErrorCode result;
3586  std::vector< Tag >::const_iterator tag_it, dst_it;
3587  std::vector< Range >::const_iterator rit;
3588  int count = 0;
3589 
3590  for( tag_it = src_tags.begin(), rit = tag_ranges.begin(); tag_it != src_tags.end(); ++tag_it, ++rit )
3591  {
3592  result = packed_tag_size( *tag_it, *rit, count );
3593  if( MB_SUCCESS != result ) return result;
3594  }
3595 
3596  // Number of tags
3597  count += sizeof( int );
3598 
3599  buff->check_space( count );
3600 
3601  PACK_INT( buff->buff_ptr, src_tags.size() );
3602 
3603  std::vector< EntityHandle > entities_vec( entities.size() );
3604  std::copy( entities.begin(), entities.end(), entities_vec.begin() );
3605 
3606  for( tag_it = src_tags.begin(), dst_it = dst_tags.begin(), rit = tag_ranges.begin(); tag_it != src_tags.end();
3607  ++tag_it, ++dst_it, ++rit )
3608  {
3609  result = pack_tag( *tag_it, *dst_it, *rit, entities_vec, buff, store_remote_handles, to_proc );
3610  if( MB_SUCCESS != result ) return result;
3611  }
3612 
3613  myDebug->tprintf( 4, "Done packing tags." );
3614 
3615  buff->set_stored_size();
3616 
3617  return MB_SUCCESS;
3618 }

References moab::Range::begin(), moab::ParallelComm::Buffer::buff_ptr, moab::ParallelComm::Buffer::check_space(), moab::Range::end(), ErrorCode, MB_SUCCESS, myDebug, moab::PACK_INT(), pack_tag(), packed_tag_size(), moab::ParallelComm::Buffer::set_stored_size(), moab::Range::size(), and moab::DebugOutput::tprintf().

Referenced by exchange_tags(), pack_buffer(), and reduce_tags().

◆ packed_tag_size()

ErrorCode moab::ParallelComm::packed_tag_size ( Tag  source_tag,
const Range &  entities,
int &  count_out 
)
private

Calculate buffer size required to pack tag data.

Parameters
source_tagThe tag for which data will be serialized
entitiesThe entities for which tag values will be serialized
count_outOutput: The required buffer size, in bytes.

Definition at line 3620 of file ParallelComm.cpp.

3621 {
3622  // For dense tags, compute size assuming all entities have that tag
3623  // For sparse tags, get number of entities w/ that tag to compute size
3624 
3625  std::vector< int > var_len_sizes;
3626  std::vector< const void* > var_len_values;
3627 
3628  // Default value
3629  count += sizeof( int );
3630  if( NULL != tag->get_default_value() ) count += tag->get_default_value_size();
3631 
3632  // Size, type, data type
3633  count += 3 * sizeof( int );
3634 
3635  // Name
3636  count += sizeof( int );
3637  count += tag->get_name().size();
3638 
3639  // Range of tag
3640  count += sizeof( int ) + tagged_entities.size() * sizeof( EntityHandle );
3641 
3642  if( tag->get_size() == MB_VARIABLE_LENGTH )
3643  {
3644  const int num_ent = tagged_entities.size();
3645  // Send a tag size for each entity
3646  count += num_ent * sizeof( int );
3647  // Send tag data for each entity
3648  var_len_sizes.resize( num_ent );
3649  var_len_values.resize( num_ent );
3650  ErrorCode result = tag->get_data( sequenceManager, errorHandler, tagged_entities, var_len_values.data(),
3651  var_len_sizes.data() );
3652  MB_CHK_SET_ERR( result, "Failed to get lenghts of variable-length tag values" );
3653  count += std::accumulate( var_len_sizes.begin(), var_len_sizes.end(), 0 );
3654  }
3655  else
3656  {
3657  // Tag data values for range or vector
3658  count += tagged_entities.size() * tag->get_size();
3659  }
3660 
3661  return MB_SUCCESS;
3662 }

References ErrorCode, errorHandler, moab::TagInfo::get_data(), moab::TagInfo::get_default_value(), moab::TagInfo::get_default_value_size(), moab::TagInfo::get_name(), moab::TagInfo::get_size(), MB_CHK_SET_ERR, MB_SUCCESS, MB_VARIABLE_LENGTH, sequenceManager, and moab::Range::size().

Referenced by pack_tags().

◆ part_tag()

Tag moab::ParallelComm::part_tag ( )
inline

Definition at line 703 of file ParallelComm.hpp.

704  {
705  return partition_tag();
706  }

References partition_tag().

Referenced by create_part().

◆ partition_sets() [1/2]

Range& moab::ParallelComm::partition_sets ( )
inline

return partition, interface set ranges

Definition at line 665 of file ParallelComm.hpp.

666  {
667  return partitionSets;
668  }

References partitionSets.

Referenced by collective_sync_partition(), create_part(), moab::ReadParallel::create_partition_sets(), moab::ReadParallel::delete_nonlocal_entities(), destroy_part(), get_part_handle(), moab::ReadParallel::load_file(), set_partitioning(), and moab::ScdInterface::tag_shared_vertices().

◆ partition_sets() [2/2]

const Range& moab::ParallelComm::partition_sets ( ) const
inline

Definition at line 669 of file ParallelComm.hpp.

670  {
671  return partitionSets;
672  }

References partitionSets.

◆ partition_tag()

Tag moab::ParallelComm::partition_tag ( )

return partitions set tag

return partition set tag

Definition at line 8294 of file ParallelComm.cpp.

8295 {
8296  if( !partitionTag )
8297  {
8298  int dum_id = -1;
8299  ErrorCode result = mbImpl->tag_get_handle( PARALLEL_PARTITION_TAG_NAME, 1, MB_TYPE_INTEGER, partitionTag,
8300  MB_TAG_SPARSE | MB_TAG_CREAT, &dum_id );
8301  if( MB_SUCCESS != result ) return 0;
8302  }
8303 
8304  return partitionTag;
8305 }

References ErrorCode, MB_SUCCESS, MB_TAG_CREAT, MB_TAG_SPARSE, MB_TYPE_INTEGER, mbImpl, PARALLEL_PARTITION_TAG_NAME, partitionTag, and moab::Interface::tag_get_handle().

Referenced by part_tag().

◆ pcomm_tag()

Tag moab::ParallelComm::pcomm_tag ( Interface *  impl,
bool  create_if_missing = true 
)
static

return pcomm tag; static because might not have a pcomm before going to look for one on the interface

return pcomm tag; passes in impl 'cuz this is a static function

Definition at line 8308 of file ParallelComm.cpp.

8309 {
8310  Tag this_tag = 0;
8311  ErrorCode result;
8312  if( create_if_missing )
8313  {
8314  result = impl->tag_get_handle( PARALLEL_COMM_TAG_NAME, MAX_SHARING_PROCS * sizeof( ParallelComm* ),
8315  MB_TYPE_OPAQUE, this_tag, MB_TAG_SPARSE | MB_TAG_CREAT );
8316  }
8317  else
8318  {
8319  result = impl->tag_get_handle( PARALLEL_COMM_TAG_NAME, MAX_SHARING_PROCS * sizeof( ParallelComm* ),
8320  MB_TYPE_OPAQUE, this_tag, MB_TAG_SPARSE );
8321  }
8322 
8323  if( MB_SUCCESS != result ) return 0;
8324 
8325  return this_tag;
8326 }

References ErrorCode, MAX_SHARING_PROCS, MB_SUCCESS, MB_TAG_CREAT, MB_TAG_SPARSE, MB_TYPE_OPAQUE, PARALLEL_COMM_TAG_NAME, and moab::Interface::tag_get_handle().

Referenced by add_pcomm(), get_all_pcomm(), get_pcomm(), remove_pcomm(), and set_partitioning().

◆ post_irecv() [1/2]

ErrorCode moab::ParallelComm::post_irecv ( std::vector< unsigned int > &  exchange_procs)

Post "MPI_Irecv" before meshing.

Parameters
exchange_procsprocessor vector exchanged

Definition at line 7023 of file ParallelComm.cpp.

7024 {
7025  // Set buffers
7026  int n_proc = exchange_procs.size();
7027  for( int i = 0; i < n_proc; i++ )
7028  get_buffers( exchange_procs[i] );
7029  reset_all_buffers();
7030 
7031  // Post ghost irecv's for entities from all communicating procs
7032  // Index requests the same as buffer/sharing procs indices
7033  int success;
7034  recvReqs.resize( 2 * buffProcs.size(), MPI_REQUEST_NULL );
7035  recvRemotehReqs.resize( 2 * buffProcs.size(), MPI_REQUEST_NULL );
7036  sendReqs.resize( 2 * buffProcs.size(), MPI_REQUEST_NULL );
7037 
7038  int incoming = 0;
7039  for( int i = 0; i < n_proc; i++ )
7040  {
7041  int ind = get_buffers( exchange_procs[i] );
7042  incoming++;
7043  PRINT_DEBUG_IRECV( procConfig.proc_rank(), buffProcs[ind], remoteOwnedBuffs[ind]->mem_ptr, INITIAL_BUFF_SIZE,
7044  MB_MESG_ENTS_SIZE, incoming );
7045  success = MPI_Irecv( remoteOwnedBuffs[ind]->mem_ptr, INITIAL_BUFF_SIZE, MPI_UNSIGNED_CHAR, buffProcs[ind],
7046  MB_MESG_ENTS_SIZE, procConfig.proc_comm(), &recvReqs[2 * ind] );
7047  if( success != MPI_SUCCESS )
7048  {
7049  MB_SET_ERR( MB_FAILURE, "Failed to post irecv in owned entity exchange" );
7050  }
7051  }
7052 
7053  return MB_SUCCESS;
7054 }

References buffProcs, get_buffers(), INITIAL_BUFF_SIZE, moab::MB_MESG_ENTS_SIZE, MB_SET_ERR, MB_SUCCESS, PRINT_DEBUG_IRECV, moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_rank(), procConfig, recvRemotehReqs, recvReqs, remoteOwnedBuffs, reset_all_buffers(), and sendReqs.

◆ post_irecv() [2/2]

ErrorCode moab::ParallelComm::post_irecv ( std::vector< unsigned int > &  shared_procs,
std::set< unsigned int > &  recv_procs 
)

Definition at line 7056 of file ParallelComm.cpp.

7057 {
7058  // Set buffers
7059  int num = shared_procs.size();
7060  for( int i = 0; i < num; i++ )
7061  get_buffers( shared_procs[i] );
7062  reset_all_buffers();
7063  num = remoteOwnedBuffs.size();
7064  for( int i = 0; i < num; i++ )
7065  remoteOwnedBuffs[i]->set_stored_size();
7066  num = localOwnedBuffs.size();
7067  for( int i = 0; i < num; i++ )
7068  localOwnedBuffs[i]->set_stored_size();
7069 
7070  // Post ghost irecv's for entities from all communicating procs
7071  // Index requests the same as buffer/sharing procs indices
7072  int success;
7073  recvReqs.resize( 2 * buffProcs.size(), MPI_REQUEST_NULL );
7074  recvRemotehReqs.resize( 2 * buffProcs.size(), MPI_REQUEST_NULL );
7075  sendReqs.resize( 2 * buffProcs.size(), MPI_REQUEST_NULL );
7076 
7077  int incoming = 0;
7078  std::set< unsigned int >::iterator it = recv_procs.begin();
7079  std::set< unsigned int >::iterator eit = recv_procs.end();
7080  for( ; it != eit; ++it )
7081  {
7082  int ind = get_buffers( *it );
7083  incoming++;
7084  PRINT_DEBUG_IRECV( procConfig.proc_rank(), buffProcs[ind], remoteOwnedBuffs[ind]->mem_ptr, INITIAL_BUFF_SIZE,
7085  MB_MESG_ENTS_SIZE, incoming );
7086  success = MPI_Irecv( remoteOwnedBuffs[ind]->mem_ptr, INITIAL_BUFF_SIZE, MPI_UNSIGNED_CHAR, buffProcs[ind],
7087  MB_MESG_ENTS_SIZE, procConfig.proc_comm(), &recvReqs[2 * ind] );
7088  if( success != MPI_SUCCESS )
7089  {
7090  MB_SET_ERR( MB_FAILURE, "Failed to post irecv in owned entity exchange" );
7091  }
7092  }
7093 
7094  return MB_SUCCESS;
7095 }

References buffProcs, get_buffers(), INITIAL_BUFF_SIZE, localOwnedBuffs, moab::MB_MESG_ENTS_SIZE, MB_SET_ERR, MB_SUCCESS, PRINT_DEBUG_IRECV, moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_rank(), procConfig, recvRemotehReqs, recvReqs, remoteOwnedBuffs, reset_all_buffers(), and sendReqs.

◆ print_buffer()

ErrorCode moab::ParallelComm::print_buffer ( unsigned char *  buff_ptr,
int  mesg_type,
int  from_proc,
bool  sent 
)
private

Definition at line 2425 of file ParallelComm.cpp.

2426 {
2427  std::cerr << procConfig.proc_rank();
2428  if( sent )
2429  std::cerr << " sent";
2430  else
2431  std::cerr << " received";
2432  std::cerr << " message type " << mesg_tag << " to/from proc " << from_proc << "; contents:" << std::endl;
2433 
2434  int msg_length, num_ents;
2435  unsigned char* orig_ptr = buff_ptr;
2436  UNPACK_INT( buff_ptr, msg_length );
2437  std::cerr << msg_length << " bytes..." << std::endl;
2438 
2439  if( MB_MESG_ENTS_SIZE == mesg_tag || MB_MESG_ENTS_LARGE == mesg_tag )
2440  {
2441  // 1. # entities = E
2442  int i, j, k;
2443  std::vector< int > ps;
2444  std::vector< EntityHandle > hs;
2445 
2446  UNPACK_INT( buff_ptr, num_ents );
2447  std::cerr << num_ents << " entities..." << std::endl;
2448 
2449  // Save place where remote handle info starts, then scan forward to ents
2450  for( i = 0; i < num_ents; i++ )
2451  {
2452  UNPACK_INT( buff_ptr, j );
2453  if( 0 > j ) return MB_FAILURE;
2454  ps.resize( j );
2455  hs.resize( j );
2456  std::cerr << "Entity " << i << ", # procs = " << j << std::endl;
2457  UNPACK_INTS( buff_ptr, ps.data(), j );
2458  UNPACK_EH( buff_ptr, hs.data(), j );
2459  std::cerr << " Procs: ";
2460  for( k = 0; k < j; k++ )
2461  std::cerr << ps[k] << " ";
2462  std::cerr << std::endl;
2463  std::cerr << " Handles: ";
2464  for( k = 0; k < j; k++ )
2465  std::cerr << hs[k] << " ";
2466  std::cerr << std::endl;
2467 
2468  if( buff_ptr - orig_ptr > msg_length )
2469  {
2470  std::cerr << "End of buffer..." << std::endl;
2471  std::cerr.flush();
2472  return MB_FAILURE;
2473  }
2474  }
2475 
2476  while( true )
2477  {
2478  EntityType this_type = MBMAXTYPE;
2479  UNPACK_TYPE( buff_ptr, this_type );
2480  assert( this_type != MBENTITYSET );
2481 
2482  // MBMAXTYPE signifies end of entities data
2483  if( MBMAXTYPE == this_type ) break;
2484 
2485  // Get the number of ents
2486  int num_ents2, verts_per_entity = 0;
2487  UNPACK_INT( buff_ptr, num_ents2 );
2488 
2489  // Unpack the nodes per entity
2490  if( MBVERTEX != this_type && num_ents2 )
2491  {
2492  UNPACK_INT( buff_ptr, verts_per_entity );
2493  }
2494 
2495  std::cerr << "Type: " << CN::EntityTypeName( this_type ) << "; num_ents = " << num_ents2;
2496  if( MBVERTEX != this_type ) std::cerr << "; verts_per_ent = " << verts_per_entity;
2497  std::cerr << std::endl;
2498  if( num_ents2 < 0 || num_ents2 > msg_length )
2499  {
2500  std::cerr << "Wrong number of entities, returning." << std::endl;
2501  return MB_FAILURE;
2502  }
2503 
2504  for( int e = 0; e < num_ents2; e++ )
2505  {
2506  // Check for existing entity, otherwise make new one
2507  if( MBVERTEX == this_type )
2508  {
2509  double coords[3];
2510  UNPACK_DBLS( buff_ptr, coords, 3 );
2511  std::cerr << "xyz = " << coords[0] << ", " << coords[1] << ", " << coords[2] << std::endl;
2512  }
2513  else
2514  {
2515  EntityHandle connect[CN::MAX_NODES_PER_ELEMENT];
2516  assert( verts_per_entity <= CN::MAX_NODES_PER_ELEMENT );
2517  UNPACK_EH( buff_ptr, connect, verts_per_entity );
2518 
2519  // Update connectivity to local handles
2520  std::cerr << "Connectivity: ";
2521  for( k = 0; k < verts_per_entity; k++ )
2522  std::cerr << connect[k] << " ";
2523  std::cerr << std::endl;
2524  }
2525 
2526  if( buff_ptr - orig_ptr > msg_length )
2527  {
2528  std::cerr << "End of buffer..." << std::endl;
2529  std::cerr.flush();
2530  return MB_FAILURE;
2531  }
2532  }
2533  }
2534  }
2535  else if( MB_MESG_REMOTEH_SIZE == mesg_tag || MB_MESG_REMOTEH_LARGE == mesg_tag )
2536  {
2537  UNPACK_INT( buff_ptr, num_ents );
2538  std::cerr << num_ents << " entities..." << std::endl;
2539  if( 0 > num_ents || num_ents > msg_length )
2540  {
2541  std::cerr << "Wrong number of entities, returning." << std::endl;
2542  return MB_FAILURE;
2543  }
2544  std::vector< EntityHandle > L1hloc( num_ents ), L1hrem( num_ents );
2545  std::vector< int > L1p( num_ents );
2546  UNPACK_INTS( buff_ptr, L1p.data(), num_ents );
2547  UNPACK_EH( buff_ptr, L1hrem.data(), num_ents );
2548  UNPACK_EH( buff_ptr, L1hloc.data(), num_ents );
2549  std::cerr << num_ents << " Entity pairs; hremote/hlocal/proc: " << std::endl;
2550  for( int i = 0; i < num_ents; i++ )
2551  {
2552  EntityType etype = TYPE_FROM_HANDLE( L1hloc[i] );
2553  std::cerr << CN::EntityTypeName( etype ) << ID_FROM_HANDLE( L1hrem[i] ) << ", "
2554  << CN::EntityTypeName( etype ) << ID_FROM_HANDLE( L1hloc[i] ) << ", " << L1p[i] << std::endl;
2555  }
2556 
2557  if( buff_ptr - orig_ptr > msg_length )
2558  {
2559  std::cerr << "End of buffer..." << std::endl;
2560  std::cerr.flush();
2561  return MB_FAILURE;
2562  }
2563  }
2564  else if( mesg_tag == MB_MESG_TAGS_SIZE || mesg_tag == MB_MESG_TAGS_LARGE )
2565  {
2566  int num_tags, dum1, data_type, tag_size;
2567  UNPACK_INT( buff_ptr, num_tags );
2568  std::cerr << "Number of tags = " << num_tags << std::endl;
2569  for( int i = 0; i < num_tags; i++ )
2570  {
2571  std::cerr << "Tag " << i << ":" << std::endl;
2572  UNPACK_INT( buff_ptr, tag_size );
2573  UNPACK_INT( buff_ptr, dum1 );
2574  UNPACK_INT( buff_ptr, data_type );
2575  std::cerr << "Tag size, type, data type = " << tag_size << ", " << dum1 << ", " << data_type << std::endl;
2576  UNPACK_INT( buff_ptr, dum1 );
2577  std::cerr << "Default value size = " << dum1 << std::endl;
2578  buff_ptr += dum1;
2579  UNPACK_INT( buff_ptr, dum1 );
2580  std::string name( (char*)buff_ptr, dum1 );
2581  std::cerr << "Tag name = " << name.c_str() << std::endl;
2582  buff_ptr += dum1;
2583  UNPACK_INT( buff_ptr, num_ents );
2584  std::cerr << "Number of ents = " << num_ents << std::endl;
2585  std::vector< EntityHandle > tmp_buff( num_ents );
2586  UNPACK_EH( buff_ptr, tmp_buff.data(), num_ents );
2587  int tot_length = 0;
2588  for( int j = 0; j < num_ents; j++ )
2589  {
2590  EntityType etype = TYPE_FROM_HANDLE( tmp_buff[j] );
2591  std::cerr << CN::EntityTypeName( etype ) << " " << ID_FROM_HANDLE( tmp_buff[j] ) << ", tag = ";
2592  if( tag_size == MB_VARIABLE_LENGTH )
2593  {
2594  UNPACK_INT( buff_ptr, dum1 );
2595  tot_length += dum1;
2596  std::cerr << "(variable, length = " << dum1 << ")" << std::endl;
2597  }
2598  else if( data_type == MB_TYPE_DOUBLE )
2599  {
2600  double dum_dbl;
2601  UNPACK_DBL( buff_ptr, dum_dbl );
2602  std::cerr << dum_dbl << std::endl;
2603  }
2604  else if( data_type == MB_TYPE_INTEGER )
2605  {
2606  int dum_int;
2607  UNPACK_INT( buff_ptr, dum_int );
2608  std::cerr << dum_int << std::endl;
2609  }
2610  else if( data_type == MB_TYPE_OPAQUE )
2611  {
2612  std::cerr << "(opaque)" << std::endl;
2613  buff_ptr += tag_size;
2614  }
2615  else if( data_type == MB_TYPE_HANDLE )
2616  {
2617  EntityHandle dum_eh;
2618  UNPACK_EH( buff_ptr, &dum_eh, 1 );
2619  std::cerr << dum_eh << std::endl;
2620  }
2621  else if( data_type == MB_TYPE_BIT )
2622  {
2623  std::cerr << "(bit)" << std::endl;
2624  buff_ptr += tag_size;
2625  }
2626  }
2627  if( tag_size == MB_VARIABLE_LENGTH ) buff_ptr += tot_length;
2628  }
2629  }
2630  else
2631  {
2632  assert( false );
2633  return MB_FAILURE;
2634  }
2635 
2636  std::cerr.flush();
2637 
2638  return MB_SUCCESS;
2639 }

References moab::CN::EntityTypeName(), moab::ID_FROM_HANDLE(), moab::CN::MAX_NODES_PER_ELEMENT, moab::MB_MESG_ENTS_LARGE, moab::MB_MESG_ENTS_SIZE, moab::MB_MESG_REMOTEH_LARGE, moab::MB_MESG_REMOTEH_SIZE, moab::MB_MESG_TAGS_LARGE, moab::MB_MESG_TAGS_SIZE, MB_SUCCESS, MB_TYPE_BIT, MB_TYPE_DOUBLE, MB_TYPE_HANDLE, MB_TYPE_INTEGER, MB_TYPE_OPAQUE, MB_VARIABLE_LENGTH, MBENTITYSET, MBMAXTYPE, MBVERTEX, moab::ProcConfig::proc_rank(), procConfig, moab::TYPE_FROM_HANDLE(), moab::UNPACK_DBL(), moab::UNPACK_DBLS(), moab::UNPACK_EH(), moab::UNPACK_INT(), moab::UNPACK_INTS(), and moab::UNPACK_TYPE().

Referenced by exchange_ghost_cells(), exchange_owned_mesh(), and recv_entities().

◆ print_debug_irecv()

void moab::ParallelComm::print_debug_irecv ( int  to,
int  from,
unsigned char *  buff,
int  size,
int  tag,
int  incoming 
)
private

Definition at line 255 of file ParallelComm.cpp.

256 {
257  myDebug->tprintf( 3, "Irecv, %d<-%d, buffer ptr = %p, tag=%d, size=%d", to, from, (void*)buff, tag, sz );
258  if( tag < MB_MESG_REMOTEH_ACK )
259  myDebug->printf( 3, ", incoming1=%d\n", incoming );
260  else if( tag < MB_MESG_TAGS_ACK )
261  myDebug->printf( 3, ", incoming2=%d\n", incoming );
262  else
263  myDebug->printf( 3, ", incoming=%d\n", incoming );
264 }

References moab::MB_MESG_REMOTEH_ACK, moab::MB_MESG_TAGS_ACK, myDebug, moab::DebugOutput::printf(), and moab::DebugOutput::tprintf().

◆ print_debug_isend()

void moab::ParallelComm::print_debug_isend ( int  from,
int  to,
unsigned char *  buff,
int  tag,
int  size 
)
private

Definition at line 250 of file ParallelComm.cpp.

251 {
252  myDebug->tprintf( 3, "Isend, %d->%d, buffer ptr = %p, tag=%d, size=%d\n", from, to, (void*)buff, tag, sz );
253 }

References myDebug, and moab::DebugOutput::tprintf().

◆ print_debug_recd()

void moab::ParallelComm::print_debug_recd ( MPI_Status  status)
private

Definition at line 266 of file ParallelComm.cpp.

267 {
268  if( myDebug->get_verbosity() == 3 )
269  {
270  int this_count;
271  int success = MPI_Get_count( &status, MPI_UNSIGNED_CHAR, &this_count );
272  if( MPI_SUCCESS != success ) this_count = -1;
273  myDebug->tprintf( 3, "Received from %d, count = %d, tag = %d\n", status.MPI_SOURCE, this_count,
274  status.MPI_TAG );
275  }
276 }

References moab::DebugOutput::get_verbosity(), myDebug, and moab::DebugOutput::tprintf().

◆ print_debug_waitany()

void moab::ParallelComm::print_debug_waitany ( std::vector< MPI_Request > &  reqs,
int  tag,
int  proc 
)
private

Definition at line 278 of file ParallelComm.cpp.

279 {
280  if( myDebug->get_verbosity() == 3 )
281  {
282  myDebug->tprintf( 3, "Waitany, p=%d, ", proc );
283  if( tag < MB_MESG_REMOTEH_ACK )
284  myDebug->print( 3, ", recv_ent_reqs=" );
285  else if( tag < MB_MESG_TAGS_ACK )
286  myDebug->print( 3, ", recv_remoteh_reqs=" );
287  else
288  myDebug->print( 3, ", recv_tag_reqs=" );
289  for( unsigned int i = 0; i < reqs.size(); i++ )
290  myDebug->printf( 3, " %p", (void*)(intptr_t)reqs[i] );
291  myDebug->print( 3, "\n" );
292  }
293 }

References moab::DebugOutput::get_verbosity(), moab::MB_MESG_REMOTEH_ACK, moab::MB_MESG_TAGS_ACK, myDebug, moab::DebugOutput::print(), moab::DebugOutput::printf(), and moab::DebugOutput::tprintf().

◆ print_pstatus() [1/2]

void moab::ParallelComm::print_pstatus ( unsigned char  pstat)

print contents of pstatus value in human-readable form to std::cut

Definition at line 9680 of file ParallelComm.cpp.

9681 {
9682  std::string str;
9683  print_pstatus( pstat, str );
9684  std::cout << str.c_str() << std::endl;
9685 }

References print_pstatus().

◆ print_pstatus() [2/2]

void moab::ParallelComm::print_pstatus ( unsigned char  pstat,
std::string &  ostr 
)

print contents of pstatus value in human-readable form

Definition at line 9657 of file ParallelComm.cpp.

9658 {
9659  std::ostringstream str;
9660  int num = 0;
9661 #define ppstat( a, b ) \
9662  { \
9663  if( pstat & ( a ) ) \
9664  { \
9665  if( num ) str << ", "; \
9666  str << ( b ); \
9667  num++; \
9668  } \
9669  }
9670 
9671  ppstat( PSTATUS_NOT_OWNED, "NOT_OWNED" );
9672  ppstat( PSTATUS_SHARED, "SHARED" );
9673  ppstat( PSTATUS_MULTISHARED, "MULTISHARED" );
9674  ppstat( PSTATUS_INTERFACE, "INTERFACE" );
9675  ppstat( PSTATUS_GHOST, "GHOST" );
9676 
9677  ostr = str.str();
9678 }

References ppstat, PSTATUS_GHOST, PSTATUS_INTERFACE, PSTATUS_MULTISHARED, PSTATUS_NOT_OWNED, and PSTATUS_SHARED.

Referenced by print_pstatus().

◆ proc_config() [1/2]

ProcConfig& moab::ParallelComm::proc_config ( )
inline

Get proc config for this communication object.

Definition at line 639 of file ParallelComm.hpp.

640  {
641  return procConfig;
642  }

References procConfig.

◆ proc_config() [2/2]

const ProcConfig& moab::ParallelComm::proc_config ( ) const
inline

Get proc config for this communication object.

Definition at line 633 of file ParallelComm.hpp.

634  {
635  return procConfig;
636  }

References procConfig.

Referenced by check_clean_iface(), moab::NCWriteGCRM::collect_mesh_info(), moab::ScdNCWriteHelper::collect_mesh_info(), moab::NCWriteHOMME::collect_mesh_info(), moab::NCWriteMPAS::collect_mesh_info(), collective_sync_partition(), comm(), moab::WriteHDF5Parallel::communicate_shared_set_data(), moab::WriteHDF5Parallel::communicate_shared_set_ids(), moab::ParCommGraph::compute_partition(), moab::WriteHDF5Parallel::create_dataset(), create_interface_sets(), moab::NCHelperDomain::create_mesh(), moab::NCHelperESMF::create_mesh(), moab::NCHelperGCRM::create_mesh(), moab::NCHelperHOMME::create_mesh(), moab::NCHelperMPAS::create_mesh(), moab::NCHelperScrip::create_mesh(), moab::WriteHDF5Parallel::create_meshset_tables(), create_part(), moab::ReadParallel::create_partition_sets(), moab::WriteHDF5Parallel::create_tag_tables(), moab::WriteHDF5Parallel::debug_barrier_line(), moab::ReadParallel::delete_nonlocal_entities(), moab::Coupler::do_normalization(), moab::WriteHDF5Parallel::exchange_file_ids(), moab::ReadHDF5::find_sets_containing(), gather_data(), moab::ParallelData::get_interface_sets(), moab::Coupler::get_matching_entities(), get_owner_handle(), get_owning_part(), get_part_handle(), get_part_id(), get_proc_nvecs(), moab::ScdInterface::get_shared_vertices(), get_sharing_parts(), iMOAB_SetDoubleTagStorageWithGid(), moab::NCHelperDomain::init_mesh_vals(), moab::NCHelperEuler::init_mesh_vals(), moab::NCHelperFV::init_mesh_vals(), moab::Coupler::initialize_tree(), moab::Coupler::interpolate(), moab::DataCoupler::interpolate(), moab::ReadParallel::load_file(), moab::ReadDamsel::load_file(), moab::Coupler::locate_points(), moab::WriteHDF5Parallel::negotiate_type_list(), pack_shared_handles(), moab::WriteHDF5Parallel::parallel_create_file(), moab::ReadHDF5::print_times(), moab::WriteHDF5Parallel::print_times(), rank(), moab::ReadHDF5::read_all_set_meta(), moab::ReadParallel::ReadParallel(), resolve_shared_ents(), resolve_shared_sets(), moab::ReadHDF5::set_up_read(), size(), moab::ScdInterface::tag_shared_vertices(), tag_shared_verts(), update_remote_data_old(), and moab::NCWriteHelper::write_set_variables().

◆ pstatus_tag()

Tag moab::ParallelComm::pstatus_tag ( )

return pstatus tag

Definition at line 8280 of file ParallelComm.cpp.

8281 {
8282  if( !pstatusTag )
8283  {
8284  unsigned char tmp_pstatus = 0;
8285  ErrorCode result = mbImpl->tag_get_handle( PARALLEL_STATUS_TAG_NAME, 1, MB_TYPE_OPAQUE, pstatusTag,
8286  MB_TAG_DENSE | MB_TAG_CREAT, &tmp_pstatus );
8287  if( MB_SUCCESS != result ) return 0;
8288  }
8289 
8290  return pstatusTag;
8291 }

References ErrorCode, MB_SUCCESS, MB_TAG_CREAT, MB_TAG_DENSE, MB_TYPE_OPAQUE, mbImpl, PARALLEL_STATUS_TAG_NAME, pstatusTag, and moab::Interface::tag_get_handle().

Referenced by assign_global_ids(), check_clean_iface(), check_sent_ents(), clean_shared_tags(), correct_thin_ghost_layers(), filter_pstatus(), get_owner_handle(), get_owning_part(), get_pstatus(), get_pstatus_entities(), get_shared_proc_tags(), get_sharing_data(), get_sharing_parts(), pack_entities(), set_pstatus_entities(), set_sharing_data(), tag_iface_entities(), and update_remote_data_old().

◆ rank()

unsigned moab::ParallelComm::rank ( ) const
inline

Definition at line 644 of file ParallelComm.hpp.

645  {
646  return proc_config().proc_rank();
647  }

References proc_config(), and moab::ProcConfig::proc_rank().

Referenced by augment_default_sets_with_ghosts(), check_my_shared_handles(), moab::ParCommGraph::compute_partition(), moab::ScdInterface::construct_box(), correct_thin_ghost_layers(), gather_data(), iMOAB_DefineTagStorage(), iMOAB_DeregisterApplication(), iMOAB_LoadMesh(), iMOAB_MergeVertices(), iMOAB_ReceiveElementTag(), iMOAB_ReceiveMesh(), iMOAB_ResolveSharedEntities(), iMOAB_SendMesh(), iMOAB_WriteLocalMesh(), moab::ReadParallel::load_file(), perform_laplacian_smoothing(), perform_lloyd_relaxation(), moab::LloydSmoother::perform_smooth(), resolve_shared_ents(), RuntimeContext::RuntimeContext(), set_sharing_data(), unpack_entities(), and update_remote_data().

◆ recv_buffer()

ErrorCode moab::ParallelComm::recv_buffer ( int  mesg_tag_expected,
const MPI_Status &  mpi_status,
Buffer *  recv_buff,
MPI_Request &  recv_2nd_req,
MPI_Request &  ack_req,
int &  this_incoming,
Buffer *  send_buff,
MPI_Request &  send_req,
MPI_Request &  sent_ack_req,
bool &  done,
Buffer *  next_buff = NULL,
int  next_tag = -1,
MPI_Request *  next_req = NULL,
int *  next_incoming = NULL 
)
private

process incoming message; if longer than the initial size, post recv for next part then send ack; if ack, send second part; else indicate that we're done and buffer is ready for processing

Definition at line 6356 of file ParallelComm.cpp.

6370 {
6371  // Process a received message; if there will be more coming,
6372  // post a receive for 2nd part then send an ack message
6373  int from_proc = mpi_status.MPI_SOURCE;
6374  int success;
6375 
6376  // Set the buff_ptr on the recv_buffer; needs to point beyond any
6377  // valid data already in the buffer
6378  recv_buff->reset_ptr( std::min( recv_buff->get_stored_size(), (int)recv_buff->alloc_size ) );
6379 
6380  if( mpi_status.MPI_TAG == mesg_tag_expected && recv_buff->get_stored_size() > (int)INITIAL_BUFF_SIZE )
6381  {
6382  // 1st message & large - allocate buffer, post irecv for 2nd message,
6383  // then send ack
6384  recv_buff->reserve( recv_buff->get_stored_size() );
6385  assert( recv_buff->alloc_size > INITIAL_BUFF_SIZE );
6386 
6387  // Will expect a 2nd message
6388  this_incoming++;
6389 
6390  PRINT_DEBUG_IRECV( procConfig.proc_rank(), from_proc, recv_buff->mem_ptr + INITIAL_BUFF_SIZE,
6391  recv_buff->get_stored_size() - INITIAL_BUFF_SIZE, mesg_tag_expected + 1, this_incoming );
6392  success = MPI_Irecv( recv_buff->mem_ptr + INITIAL_BUFF_SIZE, recv_buff->get_stored_size() - INITIAL_BUFF_SIZE,
6393  MPI_UNSIGNED_CHAR, from_proc, mesg_tag_expected + 1, procConfig.proc_comm(), &recv_req );
6394  if( success != MPI_SUCCESS )
6395  {
6396  MB_SET_ERR( MB_FAILURE, "Failed to post 2nd iRecv in ghost exchange" );
6397  }
6398 
6399  // Send ack, doesn't matter what data actually is
6400  PRINT_DEBUG_ISEND( procConfig.proc_rank(), from_proc, recv_buff->mem_ptr, mesg_tag_expected - 1,
6401  sizeof( int ) );
6402  success = MPI_Isend( recv_buff->mem_ptr, sizeof( int ), MPI_UNSIGNED_CHAR, from_proc, mesg_tag_expected - 1,
6403  procConfig.proc_comm(), &sent_ack_req );
6404  if( success != MPI_SUCCESS )
6405  {
6406  MB_SET_ERR( MB_FAILURE, "Failed to send ack in ghost exchange" );
6407  }
6408  }
6409  else if( mpi_status.MPI_TAG == mesg_tag_expected - 1 )
6410  {
6411  // Got an ack back, send the 2nd half of message
6412 
6413  // Should be a large message if we got this
6414  assert( *( (size_t*)send_buff->mem_ptr ) > INITIAL_BUFF_SIZE );
6415 
6416  // Post irecv for next message, then send 2nd message
6417  if( next_buff )
6418  {
6419  // We'll expect a return message
6420  ( *next_incoming )++;
6421  PRINT_DEBUG_IRECV( procConfig.proc_rank(), from_proc, next_buff->mem_ptr, INITIAL_BUFF_SIZE, next_tag,
6422  *next_incoming );
6423 
6424  success = MPI_Irecv( next_buff->mem_ptr, INITIAL_BUFF_SIZE, MPI_UNSIGNED_CHAR, from_proc, next_tag,
6425  procConfig.proc_comm(), next_req );
6426  if( success != MPI_SUCCESS )
6427  {
6428  MB_SET_ERR( MB_FAILURE, "Failed to post next irecv in ghost exchange" );
6429  }
6430  }
6431 
6432  // Send 2nd message
6433  PRINT_DEBUG_ISEND( procConfig.proc_rank(), from_proc, send_buff->mem_ptr + INITIAL_BUFF_SIZE,
6434  mesg_tag_expected + 1, send_buff->get_stored_size() - INITIAL_BUFF_SIZE );
6435 
6436  assert( send_buff->get_stored_size() - INITIAL_BUFF_SIZE < send_buff->alloc_size &&
6437  0 <= send_buff->get_stored_size() );
6438  success = MPI_Isend( send_buff->mem_ptr + INITIAL_BUFF_SIZE, send_buff->get_stored_size() - INITIAL_BUFF_SIZE,
6439  MPI_UNSIGNED_CHAR, from_proc, mesg_tag_expected + 1, procConfig.proc_comm(), &send_req );
6440  if( success != MPI_SUCCESS )
6441  {
6442  MB_SET_ERR( MB_FAILURE, "Failed to send 2nd message in ghost exchange" );
6443  }
6444  }
6445  else if( ( mpi_status.MPI_TAG == mesg_tag_expected && recv_buff->get_stored_size() <= (int)INITIAL_BUFF_SIZE ) ||
6446  mpi_status.MPI_TAG == mesg_tag_expected + 1 )
6447  {
6448  // Message completely received - signal that we're done
6449  done = true;
6450  }
6451 
6452  return MB_SUCCESS;
6453 }

References moab::ParallelComm::Buffer::alloc_size, moab::ParallelComm::Buffer::get_stored_size(), INITIAL_BUFF_SIZE, MB_SET_ERR, MB_SUCCESS, moab::ParallelComm::Buffer::mem_ptr, PRINT_DEBUG_IRECV, PRINT_DEBUG_ISEND, moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_rank(), procConfig, moab::ParallelComm::Buffer::reserve(), and moab::ParallelComm::Buffer::reset_ptr().

Referenced by exchange_ghost_cells(), exchange_owned_mesh(), exchange_tags(), recv_entities(), recv_messages(), recv_remote_handle_messages(), reduce_tags(), send_recv_entities(), and settle_intersection_points().

◆ recv_entities() [1/2]

ErrorCode moab::ParallelComm::recv_entities ( const int  from_proc,
const bool  store_remote_handles,
const bool  is_iface,
Range &  final_ents,
int &  incomming1,
int &  incoming2,
std::vector< std::vector< EntityHandle > > &  L1hloc,
std::vector< std::vector< EntityHandle > > &  L1hrem,
std::vector< std::vector< int > > &  L1p,
std::vector< EntityHandle > &  L2hloc,
std::vector< EntityHandle > &  L2hrem,
std::vector< unsigned int > &  L2p,
std::vector< MPI_Request > &  recv_remoteh_reqs,
bool  wait_all = true 
)

Receive entities from another processor, optionally waiting until it's done.

Receive entities from another processor, with adjs, sets, and tags. If store_remote_handles is true, this call sends back handles assigned to the entities received.

Parameters
from_procSource processor
store_remote_handlesIf true, send message with new entity handles to source processor (currently unsupported)
final_entsRange containing all entities received
incomingkeep track if any messages are coming to this processor (newly added)
wait_allIf true, wait until all messages received/sent complete

Definition at line 1093 of file ParallelComm.cpp.

1107 {
1108 #ifndef MOAB_HAVE_MPI
1109  return MB_FAILURE;
1110 #else
1111  // Non-blocking receive for the first message (having size info)
1112  int ind1 = get_buffers( from_proc );
1113  incoming1++;
1114  PRINT_DEBUG_IRECV( procConfig.proc_rank(), from_proc, remoteOwnedBuffs[ind1]->mem_ptr, INITIAL_BUFF_SIZE,
1115  MB_MESG_ENTS_SIZE, incoming1 );
1116  int success = MPI_Irecv( remoteOwnedBuffs[ind1]->mem_ptr, INITIAL_BUFF_SIZE, MPI_UNSIGNED_CHAR, from_proc,
1117  MB_MESG_ENTS_SIZE, procConfig.proc_comm(), &recvReqs[2 * ind1] );
1118  if( success != MPI_SUCCESS )
1119  {
1120  MB_SET_ERR( MB_FAILURE, "Failed to post irecv in ghost exchange" );
1121  }
1122 
1123  // Receive messages in while loop
1124  return recv_messages( from_proc, store_remote_handles, is_iface, final_ents, incoming1, incoming2, L1hloc, L1hrem,
1125  L1p, L2hloc, L2hrem, L2p, recv_remoteh_reqs );
1126 #endif
1127 }

References get_buffers(), INITIAL_BUFF_SIZE, moab::MB_MESG_ENTS_SIZE, MB_SET_ERR, PRINT_DEBUG_IRECV, moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_rank(), procConfig, recv_messages(), recvReqs, and remoteOwnedBuffs.

◆ recv_entities() [2/2]

ErrorCode moab::ParallelComm::recv_entities ( std::set< unsigned int > &  recv_procs,
int  incoming1,
int  incoming2,
const bool  store_remote_handles,
const bool  migrate = false 
)

Definition at line 1129 of file ParallelComm.cpp.

1134 {
1135  //===========================================
1136  // Receive/unpack new entities
1137  //===========================================
1138  // Number of incoming messages is the number of procs we communicate with
1139  int success, ind, i;
1140  ErrorCode result;
1141  MPI_Status status;
1142  std::vector< std::vector< EntityHandle > > recd_ents( buffProcs.size() );
1143  std::vector< std::vector< EntityHandle > > L1hloc( buffProcs.size() ), L1hrem( buffProcs.size() );
1144  std::vector< std::vector< int > > L1p( buffProcs.size() );
1145  std::vector< EntityHandle > L2hloc, L2hrem;
1146  std::vector< unsigned int > L2p;
1147  std::vector< EntityHandle > new_ents;
1148 
1149  while( incoming1 )
1150  {
1151  // Wait for all recvs of ents before proceeding to sending remote handles,
1152  // b/c some procs may have sent to a 3rd proc ents owned by me;
1153  PRINT_DEBUG_WAITANY( recvReqs, MB_MESG_ENTS_SIZE, procConfig.proc_rank() );
1154 
1155  success = MPI_Waitany( 2 * buffProcs.size(), recvReqs.data(), &ind, &status );
1156  if( MPI_SUCCESS != success )
1157  {
1158  MB_SET_ERR( MB_FAILURE, "Failed in waitany in owned entity exchange" );
1159  }
1160 
1161  PRINT_DEBUG_RECD( status );
1162 
1163  // OK, received something; decrement incoming counter
1164  incoming1--;
1165  bool done = false;
1166 
1167  // In case ind is for ack, we need index of one before it
1168  unsigned int base_ind = 2 * ( ind / 2 );
1169  result = recv_buffer( MB_MESG_ENTS_SIZE, status, remoteOwnedBuffs[ind / 2], recvReqs[ind], recvReqs[ind + 1],
1170  incoming1, localOwnedBuffs[ind / 2], sendReqs[base_ind], sendReqs[base_ind + 1], done,
1171  ( store_remote_handles ? localOwnedBuffs[ind / 2] : NULL ), MB_MESG_REMOTEH_SIZE,
1172  &recvRemotehReqs[base_ind], &incoming2 );
1173  MB_CHK_SET_ERR( result, "Failed to receive buffer" );
1174 
1175  if( done )
1176  {
1177  if( myDebug->get_verbosity() == 4 )
1178  {
1179  msgs.resize( msgs.size() + 1 );
1180  msgs.back() = new Buffer( *remoteOwnedBuffs[ind / 2] );
1181  }
1182 
1183  // Message completely received - process buffer that was sent
1184  remoteOwnedBuffs[ind / 2]->reset_ptr( sizeof( int ) );
1185  result = unpack_buffer( remoteOwnedBuffs[ind / 2]->buff_ptr, store_remote_handles, buffProcs[ind / 2],
1186  ind / 2, L1hloc, L1hrem, L1p, L2hloc, L2hrem, L2p, new_ents, true );
1187  if( MB_SUCCESS != result )
1188  {
1189  std::cout << "Failed to unpack entities. Buffer contents:" << std::endl;
1190  print_buffer( remoteOwnedBuffs[ind / 2]->mem_ptr, MB_MESG_ENTS_SIZE, buffProcs[ind / 2], false );
1191  return result;
1192  }
1193 
1194  if( recvReqs.size() != 2 * buffProcs.size() )
1195  {
1196  // Post irecv's for remote handles from new proc
1197  recvRemotehReqs.resize( 2 * buffProcs.size(), MPI_REQUEST_NULL );
1198  for( i = recvReqs.size(); i < (int)( 2 * buffProcs.size() ); i += 2 )
1199  {
1200  localOwnedBuffs[i / 2]->reset_buffer();
1201  incoming2++;
1202  PRINT_DEBUG_IRECV( procConfig.proc_rank(), buffProcs[i / 2], localOwnedBuffs[i / 2]->mem_ptr,
1203  INITIAL_BUFF_SIZE, MB_MESG_REMOTEH_SIZE, incoming2 );
1204  success = MPI_Irecv( localOwnedBuffs[i / 2]->mem_ptr, INITIAL_BUFF_SIZE, MPI_UNSIGNED_CHAR,
1205  buffProcs[i / 2], MB_MESG_REMOTEH_SIZE, procConfig.proc_comm(),
1206  &recvRemotehReqs[i] );
1207  if( success != MPI_SUCCESS )
1208  {
1209  MB_SET_ERR( MB_FAILURE, "Failed to post irecv for remote handles in ghost exchange" );
1210  }
1211  }
1212  recvReqs.resize( 2 * buffProcs.size(), MPI_REQUEST_NULL );
1213  sendReqs.resize( 2 * buffProcs.size(), MPI_REQUEST_NULL );
1214  }
1215  }
1216  }
1217 
1218  // Assign and remove newly created elements from/to receive processor
1219  result = assign_entities_part( new_ents, procConfig.proc_rank() );
1220  MB_CHK_SET_ERR( result, "Failed to assign entities to part" );
1221  if( migrate )
1222  {
1223  // result = remove_entities_part(allsent, procConfig.proc_rank());MB_CHK_SET_ERR(ressult,
1224  // "Failed to remove entities to part");
1225  }
1226 
1227  // Add requests for any new addl procs
1228  if( recvReqs.size() != 2 * buffProcs.size() )
1229  {
1230  // Shouldn't get here...
1231  MB_SET_ERR( MB_FAILURE, "Requests length doesn't match proc count in entity exchange" );
1232  }
1233 
1234 #ifdef MOAB_HAVE_MPE
1235  if( myDebug->get_verbosity() == 2 )
1236  {
1237  MPE_Log_event( ENTITIES_END, procConfig.proc_rank(), "Ending recv entities." );
1238  }
1239 #endif
1240 
1241  //===========================================
1242  // Send local handles for new entity to owner
1243  //===========================================
1244  std::set< unsigned int >::iterator it = recv_procs.begin();
1245  std::set< unsigned int >::iterator eit = recv_procs.end();
1246  for( ; it != eit; ++it )
1247  {
1248  ind = get_buffers( *it );
1249  // Reserve space on front for size and for initial buff size
1250  remoteOwnedBuffs[ind]->reset_buffer( sizeof( int ) );
1251 
1252  result = pack_remote_handles( L1hloc[ind], L1hrem[ind], L1p[ind], buffProcs[ind], remoteOwnedBuffs[ind] );
1253  MB_CHK_SET_ERR( result, "Failed to pack remote handles" );
1254  remoteOwnedBuffs[ind]->set_stored_size();
1255 
1256  if( myDebug->get_verbosity() == 4 )
1257  {
1258  msgs.resize( msgs.size() + 1 );
1259  msgs.back() = new Buffer( *remoteOwnedBuffs[ind] );
1260  }
1261  result = send_buffer( buffProcs[ind], remoteOwnedBuffs[ind], MB_MESG_REMOTEH_SIZE, sendReqs[2 * ind],
1262  recvRemotehReqs[2 * ind + 1], &ackbuff, incoming2 );
1263  MB_CHK_SET_ERR( result, "Failed to send remote handles" );
1264  }
1265 
1266  //===========================================
1267  // Process remote handles of my ghosteds
1268  //===========================================
1269  while( incoming2 )
1270  {
1271  PRINT_DEBUG_WAITANY( recvRemotehReqs, MB_MESG_REMOTEH_SIZE, procConfig.proc_rank() );
1272  success = MPI_Waitany( 2 * buffProcs.size(), recvRemotehReqs.data(), &ind, &status );
1273  if( MPI_SUCCESS != success )
1274  {
1275  MB_SET_ERR( MB_FAILURE, "Failed in waitany in owned entity exchange" );
1276  }
1277 
1278  // OK, received something; decrement incoming counter
1279  incoming2--;
1280 
1281  PRINT_DEBUG_RECD( status );
1282  bool done = false;
1283  unsigned int base_ind = 2 * ( ind / 2 );
1284  result = recv_buffer( MB_MESG_REMOTEH_SIZE, status, localOwnedBuffs[ind / 2], recvRemotehReqs[ind],
1285  recvRemotehReqs[ind + 1], incoming2, remoteOwnedBuffs[ind / 2], sendReqs[base_ind],
1286  sendReqs[base_ind + 1], done );
1287  MB_CHK_SET_ERR( result, "Failed to receive remote handles" );
1288  if( done )
1289  {
1290  // Incoming remote handles
1291  if( myDebug->get_verbosity() == 4 )
1292  {
1293  msgs.resize( msgs.size() + 1 );
1294  msgs.back() = new Buffer( *localOwnedBuffs[ind] );
1295  }
1296 
1297  localOwnedBuffs[ind / 2]->reset_ptr( sizeof( int ) );
1298  result =
1299  unpack_remote_handles( buffProcs[ind / 2], localOwnedBuffs[ind / 2]->buff_ptr, L2hloc, L2hrem, L2p );
1300  MB_CHK_SET_ERR( result, "Failed to unpack remote handles" );
1301  }
1302  }
1303 
1304 #ifdef MOAB_HAVE_MPE
1305  if( myDebug->get_verbosity() == 2 )
1306  {
1307  MPE_Log_event( RHANDLES_END, procConfig.proc_rank(), "Ending remote handles." );
1308  MPE_Log_event( OWNED_END, procConfig.proc_rank(), "Ending recv entities (still doing checks)." );
1309  }
1310 #endif
1311  myDebug->tprintf( 1, "Exiting recv_entities.\n" );
1312 
1313  return MB_SUCCESS;
1314 }

References ackbuff, assign_entities_part(), buffProcs, ErrorCode, get_buffers(), moab::DebugOutput::get_verbosity(), INITIAL_BUFF_SIZE, localOwnedBuffs, MB_CHK_SET_ERR, moab::MB_MESG_ENTS_SIZE, moab::MB_MESG_REMOTEH_SIZE, MB_SET_ERR, MB_SUCCESS, MPE_Log_event, moab::msgs, myDebug, pack_remote_handles(), print_buffer(), PRINT_DEBUG_IRECV, PRINT_DEBUG_RECD, PRINT_DEBUG_WAITANY, moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_rank(), procConfig, recv_buffer(), recvRemotehReqs, recvReqs, remoteOwnedBuffs, send_buffer(), sendReqs, moab::DebugOutput::tprintf(), unpack_buffer(), and unpack_remote_handles().

◆ recv_messages()

ErrorCode moab::ParallelComm::recv_messages ( const int  from_proc,
const bool  store_remote_handles,
const bool  is_iface,
Range &  final_ents,
int &  incoming1,
int &  incoming2,
std::vector< std::vector< EntityHandle > > &  L1hloc,
std::vector< std::vector< EntityHandle > > &  L1hrem,
std::vector< std::vector< int > > &  L1p,
std::vector< EntityHandle > &  L2hloc,
std::vector< EntityHandle > &  L2hrem,
std::vector< unsigned int > &  L2p,
std::vector< MPI_Request > &  recv_remoteh_reqs 
)

Receive messages from another processor in while loop.

Receive messages from another processor.

Parameters
from_procSource processor
store_remote_handlesIf true, send message with new entity handles to source processor (currently unsupported)
final_entsRange containing all entities received
incomingkeep track if any messages are coming to this processor (newly added)

Definition at line 1316 of file ParallelComm.cpp.

1329 {
1330 #ifndef MOAB_HAVE_MPI
1331  return MB_FAILURE;
1332 #else
1333  MPI_Status status;
1334  ErrorCode result;
1335  int ind1 = get_buffers( from_proc );
1336  int success, ind2;
1337  std::vector< EntityHandle > new_ents;
1338 
1339  // Wait and receive messages
1340  while( incoming1 )
1341  {
1342  PRINT_DEBUG_WAITANY( recvReqs, MB_MESG_TAGS_SIZE, procConfig.proc_rank() );
1343  success = MPI_Waitany( 2, &recvReqs[2 * ind1], &ind2, &status );
1344  if( MPI_SUCCESS != success )
1345  {
1346  MB_SET_ERR( MB_FAILURE, "Failed in waitany in recv_messages" );
1347  }
1348 
1349  PRINT_DEBUG_RECD( status );
1350 
1351  // OK, received something; decrement incoming counter
1352  incoming1--;
1353  bool done = false;
1354 
1355  // In case ind is for ack, we need index of one before it
1356  ind2 += 2 * ind1;
1357  unsigned int base_ind = 2 * ( ind2 / 2 );
1358 
1359  result = recv_buffer( MB_MESG_ENTS_SIZE, status, remoteOwnedBuffs[ind2 / 2],
1360  // recvbuff,
1361  recvReqs[ind2], recvReqs[ind2 + 1], incoming1, localOwnedBuffs[ind2 / 2],
1362  sendReqs[base_ind], sendReqs[base_ind + 1], done,
1363  ( !is_iface && store_remote_handles ? localOwnedBuffs[ind2 / 2] : NULL ),
1364  MB_MESG_REMOTEH_SIZE, &recv_remoteh_reqs[base_ind], &incoming2 );
1365  MB_CHK_SET_ERR( result, "Failed to receive buffer" );
1366 
1367  if( done )
1368  {
1369  // If it is done, unpack buffer
1370  remoteOwnedBuffs[ind2 / 2]->reset_ptr( sizeof( int ) );
1371  result = unpack_buffer( remoteOwnedBuffs[ind2 / 2]->buff_ptr, store_remote_handles, from_proc, ind2 / 2,
1372  L1hloc, L1hrem, L1p, L2hloc, L2hrem, L2p, new_ents );
1373  MB_CHK_SET_ERR( result, "Failed to unpack buffer in recev_messages" );
1374 
1375  std::copy( new_ents.begin(), new_ents.end(), range_inserter( final_ents ) );
1376 
1377  // Send local handles for new elements to owner
1378  // Reserve space on front for size and for initial buff size
1379  remoteOwnedBuffs[ind2 / 2]->reset_buffer( sizeof( int ) );
1380 
1381  result = pack_remote_handles( L1hloc[ind2 / 2], L1hrem[ind2 / 2], L1p[ind2 / 2], from_proc,
1382  remoteOwnedBuffs[ind2 / 2] );
1383  MB_CHK_SET_ERR( result, "Failed to pack remote handles" );
1384  remoteOwnedBuffs[ind2 / 2]->set_stored_size();
1385 
1386  result = send_buffer( buffProcs[ind2 / 2], remoteOwnedBuffs[ind2 / 2], MB_MESG_REMOTEH_SIZE, sendReqs[ind2],
1387  recv_remoteh_reqs[ind2 + 1], (int*)( localOwnedBuffs[ind2 / 2]->mem_ptr ),
1388  //&ackbuff,
1389  incoming2 );
1390  MB_CHK_SET_ERR( result, "Failed to send remote handles" );
1391  }
1392  }
1393 
1394  return MB_SUCCESS;
1395 #endif
1396 }

References buffProcs, ErrorCode, get_buffers(), localOwnedBuffs, MB_CHK_SET_ERR, moab::MB_MESG_ENTS_SIZE, moab::MB_MESG_REMOTEH_SIZE, moab::MB_MESG_TAGS_SIZE, MB_SET_ERR, MB_SUCCESS, pack_remote_handles(), PRINT_DEBUG_RECD, PRINT_DEBUG_WAITANY, moab::ProcConfig::proc_rank(), procConfig, recv_buffer(), recvReqs, remoteOwnedBuffs, send_buffer(), sendReqs, and unpack_buffer().

Referenced by recv_entities().

◆ recv_remote_handle_messages()

ErrorCode moab::ParallelComm::recv_remote_handle_messages ( const int  from_proc,
int &  incoming2,
std::vector< EntityHandle > &  L2hloc,
std::vector< EntityHandle > &  L2hrem,
std::vector< unsigned int > &  L2p,
std::vector< MPI_Request > &  recv_remoteh_reqs 
)

Definition at line 1398 of file ParallelComm.cpp.

1404 {
1405 #ifndef MOAB_HAVE_MPI
1406  return MB_FAILURE;
1407 #else
1408  MPI_Status status;
1409  ErrorCode result;
1410  int ind1 = get_buffers( from_proc );
1411  int success, ind2;
1412 
1413  while( incoming2 )
1414  {
1415  PRINT_DEBUG_WAITANY( recv_remoteh_reqs, MB_MESG_REMOTEH_SIZE, procConfig.proc_rank() );
1416  success = MPI_Waitany( 2, &recv_remoteh_reqs[2 * ind1], &ind2, &status );
1417  if( MPI_SUCCESS != success )
1418  {
1419  MB_SET_ERR( MB_FAILURE, "Failed in waitany in recv_remote_handle_messages" );
1420  }
1421 
1422  // OK, received something; decrement incoming counter
1423  incoming2--;
1424 
1425  PRINT_DEBUG_RECD( status );
1426 
1427  bool done = false;
1428  ind2 += 2 * ind1;
1429  unsigned int base_ind = 2 * ( ind2 / 2 );
1430  result = recv_buffer( MB_MESG_REMOTEH_SIZE, status, localOwnedBuffs[ind2 / 2], recv_remoteh_reqs[ind2],
1431  recv_remoteh_reqs[ind2 + 1], incoming2, remoteOwnedBuffs[ind2 / 2], sendReqs[base_ind],
1432  sendReqs[base_ind + 1], done );
1433  MB_CHK_SET_ERR( result, "Failed to receive remote handles" );
1434  if( done )
1435  {
1436  // Incoming remote handles
1437  localOwnedBuffs[ind2 / 2]->reset_ptr( sizeof( int ) );
1438  result =
1439  unpack_remote_handles( buffProcs[ind2 / 2], localOwnedBuffs[ind2 / 2]->buff_ptr, L2hloc, L2hrem, L2p );
1440  MB_CHK_SET_ERR( result, "Failed to unpack remote handles" );
1441  }
1442  }
1443 
1444  return MB_SUCCESS;
1445 #endif
1446 }

References buffProcs, ErrorCode, get_buffers(), localOwnedBuffs, MB_CHK_SET_ERR, moab::MB_MESG_REMOTEH_SIZE, MB_SET_ERR, MB_SUCCESS, PRINT_DEBUG_RECD, PRINT_DEBUG_WAITANY, moab::ProcConfig::proc_rank(), procConfig, recv_buffer(), remoteOwnedBuffs, sendReqs, and unpack_remote_handles().

◆ reduce()

template<class T >
ErrorCode moab::ParallelComm::reduce ( const MPI_Op  mpi_op,
int  num_ents,
void *  old_vals,
void *  new_vals 
)
private

Definition at line 3963 of file ParallelComm.cpp.

3964 {
3965  T* old_tmp = reinterpret_cast< T* >( old_vals );
3966  // T *new_tmp = reinterpret_cast<T*>(new_vals);
3967  // new vals pointer needs to be aligned , some compilers will optimize and will shift
3968 
3969  std::vector< T > new_values;
3970  new_values.resize( num_ents );
3971  memcpy( new_values.data(), new_vals, num_ents * sizeof( T ) );
3972  T* new_tmp = new_values.data();
3973 
3974  if( mpi_op == MPI_SUM )
3975  std::transform( old_tmp, old_tmp + num_ents, new_tmp, new_tmp, ADD< T > );
3976  else if( mpi_op == MPI_PROD )
3977  std::transform( old_tmp, old_tmp + num_ents, new_tmp, new_tmp, MULT< T > );
3978  else if( mpi_op == MPI_MAX )
3979  std::transform( old_tmp, old_tmp + num_ents, new_tmp, new_tmp, MAX< T > );
3980  else if( mpi_op == MPI_MIN )
3981  std::transform( old_tmp, old_tmp + num_ents, new_tmp, new_tmp, MIN< T > );
3982  else if( mpi_op == MPI_LAND )
3983  std::transform( old_tmp, old_tmp + num_ents, new_tmp, new_tmp, LAND< T > );
3984  else if( mpi_op == MPI_LOR )
3985  std::transform( old_tmp, old_tmp + num_ents, new_tmp, new_tmp, LOR< T > );
3986  else if( mpi_op == MPI_LXOR )
3987  std::transform( old_tmp, old_tmp + num_ents, new_tmp, new_tmp, LXOR< T > );
3988  else if( mpi_op == MPI_BAND || mpi_op == MPI_BOR || mpi_op == MPI_BXOR )
3989  {
3990  std::cerr << "Bitwise operations not allowed in tag reductions." << std::endl;
3991  return MB_FAILURE;
3992  }
3993  else if( mpi_op != MPI_OP_NULL )
3994  {
3995  std::cerr << "Unknown MPI operation type." << std::endl;
3996  return MB_TYPE_OUT_OF_RANGE;
3997  }
3998 
3999  // copy now the result back where it should be
4000  memcpy( new_vals, new_tmp, num_ents * sizeof( T ) );
4001  std::vector< T >().swap( new_values ); // way to release allocated vector
4002 
4003  return MB_SUCCESS;
4004 }

References MB_SUCCESS, and MB_TYPE_OUT_OF_RANGE.

◆ reduce_tags() [1/3]

ErrorCode moab::ParallelComm::reduce_tags ( const char *  tag_name,
const MPI_Op  mpi_op,
const Range &  entities 
)
inline

Perform data reduction operation for all shared and ghosted entities Same as std::vector variant except for one tag specified by name.

Parameters
tag_nameName of tag to be reduced
mpi_opOperation type
entitiesEntities on which reduction will be made; if empty, operates on all shared entities

Definition at line 1611 of file ParallelComm.hpp.

1612 {
1613  // get the tag handle
1614  std::vector< Tag > tags( 1 );
1615  ErrorCode result = mbImpl->tag_get_handle( tag_name, 0, MB_TYPE_OPAQUE, tags[0], MB_TAG_ANY );
1616  if( MB_SUCCESS != result )
1617  return result;
1618  else if( !tags[0] )
1619  return MB_TAG_NOT_FOUND;
1620 
1621  return reduce_tags( tags, tags, mpi_op, entities );
1622 }

References ErrorCode, MB_SUCCESS, MB_TAG_ANY, MB_TAG_NOT_FOUND, MB_TYPE_OPAQUE, mbImpl, reduce_tags(), and moab::Interface::tag_get_handle().

◆ reduce_tags() [2/3]

ErrorCode moab::ParallelComm::reduce_tags ( const std::vector< Tag > &  src_tags,
const std::vector< Tag > &  dst_tags,
const MPI_Op  mpi_op,
const Range &  entities 
)

Perform data reduction operation for all shared and ghosted entities This function should be called collectively over the communicator for this ParallelComm. If this version is called, all ghosted/shared entities should have a value for this tag (or the tag should have a default value). Operation is any MPI_Op, with result stored in destination tag.

Parameters
src_tagsVector of tag handles to be reduced
dst_tagsVector of tag handles in which the answer will be stored
mpi_opOperation type
entitiesEntities on which reduction will be made; if empty, operates on all shared entities

Definition at line 8020 of file ParallelComm.cpp.

8024 {
8025  ErrorCode result;
8026  int success;
8027 
8028  myDebug->tprintf( 1, "Entering reduce_tags\n" );
8029 
8030  // Check that restrictions are met: number of source/dst tags...
8031  if( src_tags.size() != dst_tags.size() )
8032  {
8033  MB_SET_ERR( MB_FAILURE, "Source and destination tag handles must be specified for reduce_tags" );
8034  }
8035 
8036  // ... tag data types
8037  std::vector< Tag >::const_iterator vits, vitd;
8038  int tags_size, tagd_size;
8039  DataType tags_type, tagd_type;
8040  std::vector< unsigned char > vals;
8041  std::vector< int > tags_sizes;
8042  for( vits = src_tags.begin(), vitd = dst_tags.begin(); vits != src_tags.end(); ++vits, ++vitd )
8043  {
8044  // Checks on tag characteristics
8045  result = mbImpl->tag_get_data_type( *vits, tags_type );
8046  MB_CHK_SET_ERR( result, "Failed to get src tag data type" );
8047  if( tags_type != MB_TYPE_INTEGER && tags_type != MB_TYPE_DOUBLE && tags_type != MB_TYPE_BIT )
8048  {
8049  MB_SET_ERR( MB_FAILURE, "Src/dst tags must have integer, double, or bit data type" );
8050  }
8051 
8052  result = mbImpl->tag_get_bytes( *vits, tags_size );
8053  MB_CHK_SET_ERR( result, "Failed to get src tag bytes" );
8054  vals.resize( tags_size );
8055  result = mbImpl->tag_get_default_value( *vits, vals.data() );
8056  MB_CHK_SET_ERR( result, "Src tag must have default value" );
8057 
8058  tags_sizes.push_back( tags_size );
8059 
8060  // OK, those passed; now check whether dest tags, if specified, agree with src tags
8061  if( *vits == *vitd ) continue;
8062 
8063  result = mbImpl->tag_get_bytes( *vitd, tagd_size );
8064  MB_CHK_SET_ERR( result, "Coudln't get dst tag bytes" );
8065  if( tags_size != tagd_size )
8066  {
8067  MB_SET_ERR( MB_FAILURE, "Sizes between src and dst tags don't match" );
8068  }
8069  result = mbImpl->tag_get_data_type( *vitd, tagd_type );
8070  MB_CHK_SET_ERR( result, "Coudln't get dst tag data type" );
8071  if( tags_type != tagd_type )
8072  {
8073  MB_SET_ERR( MB_FAILURE, "Src and dst tags must be of same data type" );
8074  }
8075  }
8076 
8077  // Get all procs interfacing to this proc
8078  std::set< unsigned int > exch_procs;
8079  result = get_comm_procs( exch_procs );
8080 
8081  // Post ghost irecv's for all interface procs
8082  // Index requests the same as buffer/sharing procs indices
8083  std::vector< MPI_Request > recv_tag_reqs( 3 * buffProcs.size(), MPI_REQUEST_NULL );
8084 
8085  std::vector< unsigned int >::iterator sit;
8086  int ind;
8087 
8088  reset_all_buffers();
8089  int incoming = 0;
8090 
8091  for( ind = 0, sit = buffProcs.begin(); sit != buffProcs.end(); ++sit, ind++ )
8092  {
8093  incoming++;
8094  PRINT_DEBUG_IRECV( *sit, procConfig.proc_rank(), remoteOwnedBuffs[ind]->mem_ptr, INITIAL_BUFF_SIZE,
8095  MB_MESG_TAGS_SIZE, incoming );
8096 
8097  success = MPI_Irecv( remoteOwnedBuffs[ind]->mem_ptr, INITIAL_BUFF_SIZE, MPI_UNSIGNED_CHAR, *sit,
8098  MB_MESG_TAGS_SIZE, procConfig.proc_comm(), &recv_tag_reqs[3 * ind] );
8099  if( success != MPI_SUCCESS )
8100  {
8101  MB_SET_ERR( MB_FAILURE, "Failed to post irecv in ghost exchange" );
8102  }
8103  }
8104 
8105  // Pack and send tags from this proc to others
8106  // Make sendReqs vector to simplify initialization
8107  sendReqs.resize( 3 * buffProcs.size(), MPI_REQUEST_NULL );
8108 
8109  // Take all shared entities if incoming list is empty
8110  Range entities;
8111  if( entities_in.empty() )
8112  std::copy( sharedEnts.begin(), sharedEnts.end(), range_inserter( entities ) );
8113  else
8114  entities = entities_in;
8115 
8116  // If the tags are different, copy the source to the dest tag locally
8117  std::vector< Tag >::const_iterator vit = src_tags.begin(), vit2 = dst_tags.begin();
8118  std::vector< int >::const_iterator vsizes = tags_sizes.begin();
8119  for( ; vit != src_tags.end(); ++vit, ++vit2, ++vsizes )
8120  {
8121  if( *vit == *vit2 ) continue;
8122  vals.resize( entities.size() * ( *vsizes ) );
8123  result = mbImpl->tag_get_data( *vit, entities, vals.data() );
8124  MB_CHK_SET_ERR( result, "Didn't get data properly" );
8125  result = mbImpl->tag_set_data( *vit2, entities, vals.data() );
8126  MB_CHK_SET_ERR( result, "Didn't set data properly" );
8127  }
8128 
8129  int dum_ack_buff;
8130 
8131  for( ind = 0, sit = buffProcs.begin(); sit != buffProcs.end(); ++sit, ind++ )
8132  {
8133  Range tag_ents = entities;
8134 
8135  // Get ents shared by proc *sit
8136  result = filter_pstatus( tag_ents, PSTATUS_SHARED, PSTATUS_AND, *sit );
8137  MB_CHK_SET_ERR( result, "Failed pstatus AND check" );
8138 
8139  // Pack-send
8140  std::vector< Range > tag_ranges;
8141  for( vit = src_tags.begin(); vit != src_tags.end(); ++vit )
8142  {
8143  const void* ptr;
8144  int sz;
8145  if( mbImpl->tag_get_default_value( *vit, ptr, sz ) != MB_SUCCESS )
8146  {
8147  Range tagged_ents;
8148  mbImpl->get_entities_by_type_and_tag( 0, MBMAXTYPE, &*vit, 0, 1, tagged_ents );
8149  tag_ranges.push_back( intersect( tag_ents, tagged_ents ) );
8150  }
8151  else
8152  tag_ranges.push_back( tag_ents );
8153  }
8154 
8155  // Pack the data
8156  // Reserve space on front for size and for initial buff size
8157  localOwnedBuffs[ind]->reset_ptr( sizeof( int ) );
8158 
8159  result = pack_tags( tag_ents, src_tags, dst_tags, tag_ranges, localOwnedBuffs[ind], true, *sit );
8160  MB_CHK_SET_ERR( result, "Failed to count buffer in pack_send_tag" );
8161 
8162  // Now send it
8163  result = send_buffer( *sit, localOwnedBuffs[ind], MB_MESG_TAGS_SIZE, sendReqs[3 * ind],
8164  recv_tag_reqs[3 * ind + 2], &dum_ack_buff, incoming );
8165  MB_CHK_SET_ERR( result, "Failed to send buffer" );
8166  }
8167 
8168  // Receive/unpack tags
8169  while( incoming )
8170  {
8171  MPI_Status status;
8172  int index_in_recv_requests;
8173  PRINT_DEBUG_WAITANY( recv_tag_reqs, MB_MESG_TAGS_SIZE, procConfig.proc_rank() );
8174  success = MPI_Waitany( 3 * buffProcs.size(), recv_tag_reqs.data(), &index_in_recv_requests, &status );
8175  if( MPI_SUCCESS != success )
8176  {
8177  MB_SET_ERR( MB_FAILURE, "Failed in waitany in ghost exchange" );
8178  }
8179  ind = index_in_recv_requests / 3;
8180 
8181  PRINT_DEBUG_RECD( status );
8182 
8183  // OK, received something; decrement incoming counter
8184  incoming--;
8185 
8186  bool done = false;
8187  std::vector< EntityHandle > dum_vec;
8188  result = recv_buffer( MB_MESG_TAGS_SIZE, status, remoteOwnedBuffs[ind],
8189  recv_tag_reqs[3 * ind + 1], // This is for receiving the second message
8190  recv_tag_reqs[3 * ind + 2], // This would be for ack, but it is not
8191  // used; consider removing it
8192  incoming, localOwnedBuffs[ind],
8193  sendReqs[3 * ind + 1], // Send request for sending the second message
8194  sendReqs[3 * ind + 2], // This is for sending the ack
8195  done );
8196  MB_CHK_SET_ERR( result, "Failed to resize recv buffer" );
8197  if( done )
8198  {
8199  remoteOwnedBuffs[ind]->reset_ptr( sizeof( int ) );
8200  result = unpack_tags( remoteOwnedBuffs[ind]->buff_ptr, dum_vec, true, buffProcs[ind], &mpi_op );
8201  MB_CHK_SET_ERR( result, "Failed to recv-unpack-tag message" );
8202  }
8203  }
8204 
8205  // OK, now wait
8206  if( myDebug->get_verbosity() == 5 )
8207  {
8208  success = MPI_Barrier( procConfig.proc_comm() );
8209  }
8210  else
8211  {
8212  MPI_Status status[3 * MAX_SHARING_PROCS];
8213  success = MPI_Waitall( 3 * buffProcs.size(), sendReqs.data(), status );
8214  }
8215  if( MPI_SUCCESS != success )
8216  {
8217  MB_SET_ERR( MB_FAILURE, "Failure in waitall in tag exchange" );
8218  }
8219 
8220  myDebug->tprintf( 1, "Exiting reduce_tags" );
8221 
8222  return MB_SUCCESS;
8223 }

References moab::Range::begin(), buffProcs, moab::Range::empty(), ErrorCode, filter_pstatus(), get_comm_procs(), moab::Interface::get_entities_by_type_and_tag(), moab::DebugOutput::get_verbosity(), INITIAL_BUFF_SIZE, moab::intersect(), localOwnedBuffs, MAX_SHARING_PROCS, MB_CHK_SET_ERR, moab::MB_MESG_TAGS_SIZE, MB_SET_ERR, MB_SUCCESS, MB_TYPE_BIT, MB_TYPE_DOUBLE, MB_TYPE_INTEGER, mbImpl, MBMAXTYPE, myDebug, pack_tags(), PRINT_DEBUG_IRECV, PRINT_DEBUG_RECD, PRINT_DEBUG_WAITANY, moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_rank(), procConfig, PSTATUS_AND, PSTATUS_SHARED, recv_buffer(), remoteOwnedBuffs, reset_all_buffers(), send_buffer(), sendReqs, sharedEnts, moab::Range::size(), moab::Interface::tag_get_bytes(), moab::Interface::tag_get_data(), moab::Interface::tag_get_data_type(), moab::Interface::tag_get_default_value(), moab::Interface::tag_set_data(), moab::DebugOutput::tprintf(), and unpack_tags().

Referenced by iMOAB_ReduceTagsMax(), main(), and reduce_tags().

◆ reduce_tags() [3/3]

ErrorCode moab::ParallelComm::reduce_tags ( Tag  tag_handle,
const MPI_Op  mpi_op,
const Range &  entities 
)
inline

Perform data reduction operation for all shared and ghosted entities Same as std::vector variant except for one tag specified by handle.

Parameters
tag_nameName of tag to be reduced
mpi_opOperation type
entitiesEntities on which reduction will be made; if empty, operates on all shared entities

Definition at line 1624 of file ParallelComm.hpp.

1625 {
1626  // get the tag handle
1627  std::vector< Tag > tags;
1628  tags.push_back( tagh );
1629 
1630  return reduce_tags( tags, tags, mpi_op, entities );
1631 }

References reduce_tags().

◆ reduce_void()

ErrorCode moab::ParallelComm::reduce_void ( int  tag_data_type,
const MPI_Op  mpi_op,
int  num_ents,
void *  old_vals,
void *  new_vals 
)
private

Definition at line 4006 of file ParallelComm.cpp.

4011 {
4012  ErrorCode result;
4013  switch( tag_data_type )
4014  {
4015  case MB_TYPE_INTEGER:
4016  result = reduce< int >( mpi_op, num_ents, old_vals, new_vals );
4017  break;
4018  case MB_TYPE_DOUBLE:
4019  result = reduce< double >( mpi_op, num_ents, old_vals, new_vals );
4020  break;
4021  case MB_TYPE_BIT:
4022  result = reduce< unsigned char >( mpi_op, num_ents, old_vals, new_vals );
4023  break;
4024  default:
4025  result = MB_SUCCESS;
4026  break;
4027  }
4028 
4029  return result;
4030 }

References ErrorCode, MB_SUCCESS, MB_TYPE_BIT, MB_TYPE_DOUBLE, and MB_TYPE_INTEGER.

Referenced by unpack_tags().

◆ remove_entities_part()

ErrorCode moab::ParallelComm::remove_entities_part ( Range &  entities,
const int  proc 
)
private

remove entities to the input processor part

Definition at line 7585 of file ParallelComm.cpp.

7586 {
7587  EntityHandle part_set;
7588  ErrorCode result = get_part_handle( proc, part_set );
7589  MB_CHK_SET_ERR( result, "Failed to get part handle" );
7590 
7591  if( part_set > 0 )
7592  {
7593  result = mbImpl->remove_entities( part_set, entities );
7594  MB_CHK_SET_ERR( result, "Failed to remove entities from part set" );
7595  }
7596 
7597  return MB_SUCCESS;
7598 }

References ErrorCode, get_part_handle(), MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, and moab::Interface::remove_entities().

Referenced by exchange_owned_mesh().

◆ remove_pcomm()

void moab::ParallelComm::remove_pcomm ( ParallelComm *  pc)
private

remove a pc from the iface instance tag PARALLEL_COMM

Definition at line 400 of file ParallelComm.cpp.

401 {
402  // Remove this pcomm from instance tag
403  std::vector< ParallelComm* > pc_array( MAX_SHARING_PROCS );
404  Tag pc_tag = pcomm_tag( mbImpl, true );
405 
406  const EntityHandle root = 0;
407  ErrorCode result = mbImpl->tag_get_data( pc_tag, &root, 1, (void*)pc_array.data() );
408  std::vector< ParallelComm* >::iterator pc_it = std::find( pc_array.begin(), pc_array.end(), pc );
409  assert( MB_SUCCESS == result && pc_it != pc_array.end() );
410  // Empty if test to get around compiler warning about unused var
411  if( MB_SUCCESS == result )
412  {
413  }
414 
415  *pc_it = NULL;
416  mbImpl->tag_set_data( pc_tag, &root, 1, (void*)pc_array.data() );
417 }

References ErrorCode, MAX_SHARING_PROCS, MB_SUCCESS, mbImpl, pcomm_tag(), moab::Interface::tag_get_data(), and moab::Interface::tag_set_data().

Referenced by ~ParallelComm().

◆ reset_all_buffers()

void moab::ParallelComm::reset_all_buffers ( )
inline

reset message buffers to their initial state

Definition at line 1548 of file ParallelComm.hpp.

1549 {
1550  std::vector< Buffer* >::iterator vit;
1551  for( vit = localOwnedBuffs.begin(); vit != localOwnedBuffs.end(); ++vit )
1552  ( *vit )->reset_buffer();
1553  for( vit = remoteOwnedBuffs.begin(); vit != remoteOwnedBuffs.end(); ++vit )
1554  ( *vit )->reset_buffer();
1555 }

References localOwnedBuffs, and remoteOwnedBuffs.

Referenced by exchange_ghost_cells(), exchange_owned_mesh(), exchange_tags(), post_irecv(), reduce_tags(), send_recv_entities(), and settle_intersection_points().

◆ resolve_shared_ents() [1/3]

ErrorCode moab::ParallelComm::resolve_shared_ents ( EntityHandle  this_set,
int  resolve_dim = 3,
int  shared_dim = -1,
const Tag *  id_tag = 0 
)

Resolve shared entities between processors.

Same as resolve_shared_ents(Range&), except works for all entities in instance of dimension dim.

If shared_dim is input as -1 or not input, a value one less than the maximum dimension of entities is used.

Parameters
dimDimension of entities in the partition
shared_dimMaximum dimension of shared entities to look for

Definition at line 4032 of file ParallelComm.cpp.

4033 {
4034  ErrorCode result;
4035  Range proc_ents;
4036 
4037  // Check for structured mesh, and do it differently if it is
4038  ScdInterface* scdi;
4039  result = mbImpl->query_interface( scdi );
4040  if( scdi )
4041  {
4042  result = scdi->tag_shared_vertices( this, this_set );
4043  if( MB_SUCCESS == result )
4044  {
4045  myDebug->tprintf( 1, "Total number of shared entities = %lu.\n", (unsigned long)sharedEnts.size() );
4046  return result;
4047  }
4048  }
4049 
4050  if( 0 == this_set )
4051  {
4052  // Get the entities in the partition sets
4053  for( Range::iterator rit = partitionSets.begin(); rit != partitionSets.end(); ++rit )
4054  {
4055  Range tmp_ents;
4056  result = mbImpl->get_entities_by_handle( *rit, tmp_ents, true );
4057  if( MB_SUCCESS != result ) return result;
4058  proc_ents.merge( tmp_ents );
4059  }
4060  }
4061  else
4062  {
4063  result = mbImpl->get_entities_by_handle( this_set, proc_ents, true );
4064  if( MB_SUCCESS != result ) return result;
4065  }
4066 
4067  // Resolve dim is maximal dim of entities in proc_ents
4068  if( -1 == resolve_dim )
4069  {
4070  if( !proc_ents.empty() ) resolve_dim = mbImpl->dimension_from_handle( *proc_ents.rbegin() );
4071  }
4072 
4073  // proc_ents should all be of same dimension
4074  if( resolve_dim > shared_dim &&
4075  mbImpl->dimension_from_handle( *proc_ents.rbegin() ) != mbImpl->dimension_from_handle( *proc_ents.begin() ) )
4076  {
4077  Range::iterator lower = proc_ents.lower_bound( CN::TypeDimensionMap[0].first ),
4078  upper = proc_ents.upper_bound( CN::TypeDimensionMap[resolve_dim - 1].second );
4079  proc_ents.erase( lower, upper );
4080  }
4081 
4082  // Must call even if we don't have any entities, to make sure
4083  // collective comm'n works
4084  return resolve_shared_ents( this_set, proc_ents, resolve_dim, shared_dim, NULL, id_tag );
4085 }

References moab::Range::begin(), moab::Interface::dimension_from_handle(), moab::Range::empty(), moab::Range::end(), moab::Range::erase(), ErrorCode, moab::GeomUtil::first(), moab::Interface::get_entities_by_handle(), moab::Range::lower_bound(), MB_SUCCESS, mbImpl, moab::Range::merge(), myDebug, partitionSets, moab::Interface::query_interface(), moab::Range::rbegin(), resolve_shared_ents(), sharedEnts, moab::ScdInterface::tag_shared_vertices(), moab::DebugOutput::tprintf(), moab::CN::TypeDimensionMap, and moab::Range::upper_bound().

◆ resolve_shared_ents() [2/3]

ErrorCode moab::ParallelComm::resolve_shared_ents ( EntityHandle  this_set,
Range &  proc_ents,
int  resolve_dim = -1,
int  shared_dim = -1,
Range *  skin_ents = NULL,
const Tag *  id_tag = 0 
)

Resolve shared entities between processors.

Resolve shared entities between processors for entities in proc_ents, by comparing global id tag values on vertices on skin of elements in proc_ents. Shared entities are assigned a tag that's either PARALLEL_SHARED_PROC_TAG_NAME, which is 1 integer in length, or PARALLEL_SHARED_PROCS_TAG_NAME, whose length depends on the maximum number of sharing processors. Values in these tags denote the ranks of sharing processors, and the list ends with the value -1.

If shared_dim is input as -1 or not input, a value one less than the maximum dimension of entities in proc_ents is used.

Parameters
proc_entsEntities for which to resolve shared entities
shared_dimMaximum dimension of shared entities to look for

Definition at line 4087 of file ParallelComm.cpp.

4093 {
4094 #ifdef MOAB_HAVE_MPE
4095  if( myDebug->get_verbosity() == 2 )
4096  {
4097  define_mpe();
4098  MPE_Log_event( RESOLVE_START, procConfig.proc_rank(), "Entering resolve_shared_ents." );
4099  }
4100 #endif
4101 
4102  ErrorCode result;
4103  myDebug->tprintf( 1, "Resolving shared entities.\n" );
4104 
4105  if( resolve_dim < shared_dim )
4106  {
4107  MB_SET_ERR( MB_FAILURE, "MOAB does not support vertex-based partitions, only element-based ones" );
4108  }
4109 
4110  if( -1 == shared_dim )
4111  {
4112  if( !proc_ents.empty() )
4113  shared_dim = mbImpl->dimension_from_handle( *proc_ents.begin() ) - 1;
4114  else if( resolve_dim == 3 )
4115  shared_dim = 2;
4116  }
4117  int max_global_resolve_dim = -1;
4118  int err = MPI_Allreduce( &resolve_dim, &max_global_resolve_dim, 1, MPI_INT, MPI_MAX, proc_config().proc_comm() );
4119  if( MPI_SUCCESS != err )
4120  {
4121  MB_SET_ERR( MB_FAILURE, "Unable to guess global resolve_dim" );
4122  }
4123  if( shared_dim < 0 || resolve_dim < 0 )
4124  {
4125  // MB_SET_ERR(MB_FAILURE, "Unable to guess shared_dim or resolve_dim");
4126  resolve_dim = max_global_resolve_dim;
4127  shared_dim = resolve_dim - 1;
4128  }
4129 
4130  if( resolve_dim < 0 || shared_dim < 0 ) return MB_SUCCESS;
4131  // no task has any mesh, get out
4132 
4133  // Get the skin entities by dimension
4134  Range tmp_skin_ents[4];
4135 
4136  // Get the entities to be skinned
4137  // Find the skin
4138  int skin_dim = resolve_dim - 1;
4139  if( !skin_ents )
4140  {
4141  skin_ents = tmp_skin_ents;
4142  skin_ents[resolve_dim] = proc_ents;
4143  Skinner skinner( mbImpl );
4144  result =
4145  skinner.find_skin( this_set, skin_ents[skin_dim + 1], false, skin_ents[skin_dim], NULL, true, true, true );
4146  MB_CHK_SET_ERR( result, "Failed to find skin" );
4147  myDebug->tprintf( 1, "Found skin: skin_dim: %d resolve_dim: %d , now resolving.\n", skin_dim, resolve_dim );
4148  myDebug->tprintf( 3, "skin_ents[0].size(): %d skin_ents[1].size(): %d \n", (int)skin_ents[0].size(),
4149  (int)skin_ents[1].size() );
4150  // Get entities adjacent to skin ents from shared_dim down to zero
4151  for( int this_dim = skin_dim - 1; this_dim >= 0; this_dim-- )
4152  {
4153  result =
4154  mbImpl->get_adjacencies( skin_ents[skin_dim], this_dim, true, skin_ents[this_dim], Interface::UNION );
4155  MB_CHK_SET_ERR( result, "Failed to get skin adjacencies" );
4156 
4157  if( this_set && skin_dim == 2 && this_dim == 1 )
4158  {
4159  result = mbImpl->add_entities( this_set, skin_ents[this_dim] );
4160  MB_CHK_ERR( result );
4161  }
4162  }
4163  }
4164  else if( skin_ents[resolve_dim].empty() )
4165  skin_ents[resolve_dim] = proc_ents;
4166 
4167  // Global id tag
4168  Tag gid_tag;
4169  if( id_tag )
4170  gid_tag = *id_tag;
4171  else
4172  {
4173  bool tag_created = false;
4174  int def_val = -1;
4175  result = mbImpl->tag_get_handle( GLOBAL_ID_TAG_NAME, 1, MB_TYPE_INTEGER, gid_tag, MB_TAG_DENSE | MB_TAG_CREAT,
4176  &def_val, &tag_created );
4177  if( MB_ALREADY_ALLOCATED != result && MB_SUCCESS != result )
4178  {
4179  MB_SET_ERR( result, "Failed to create/get gid tag handle" );
4180  }
4181  else if( tag_created )
4182  {
4183  // Just created it, so we need global ids
4184  result = assign_global_ids( this_set, skin_dim + 1, true, true, true );
4185  MB_CHK_SET_ERR( result, "Failed to assign global ids" );
4186  }
4187  }
4188 
4189  DataType tag_type;
4190  result = mbImpl->tag_get_data_type( gid_tag, tag_type );
4191  MB_CHK_SET_ERR( result, "Failed to get tag data type" );
4192  int bytes_per_tag;
4193  result = mbImpl->tag_get_bytes( gid_tag, bytes_per_tag );
4194  MB_CHK_SET_ERR( result, "Failed to get number of bytes per tag" );
4195  // On 64 bits, long and int are different
4196  // On 32 bits, they are not; if size of long is 8, it is a 64 bit machine (really?)
4197 
4198  // Get gids for skin ents in a vector, to pass to gs
4199  std::vector< long > lgid_data( skin_ents[0].size() );
4200  // Size is either long or int
4201  // On 64 bit is 8 or 4
4202  if( sizeof( long ) == bytes_per_tag && ( ( MB_TYPE_HANDLE == tag_type ) || ( MB_TYPE_OPAQUE == tag_type ) ) )
4203  { // It is a special id tag
4204  result = mbImpl->tag_get_data( gid_tag, skin_ents[0], lgid_data.data() );
4205  MB_CHK_SET_ERR( result, "Couldn't get gid tag for skin vertices" );
4206  }
4207  else if( 4 == bytes_per_tag )
4208  { // Must be GLOBAL_ID tag or 32 bits ...
4209  std::vector< int > gid_data( lgid_data.size() );
4210  result = mbImpl->tag_get_data( gid_tag, skin_ents[0], gid_data.data() );
4211  MB_CHK_SET_ERR( result, "Failed to get gid tag for skin vertices" );
4212  std::copy( gid_data.begin(), gid_data.end(), lgid_data.begin() );
4213  }
4214  else
4215  {
4216  // Not supported flag
4217  MB_SET_ERR( MB_FAILURE, "Unsupported id tag" );
4218  }
4219 
4220  // Put handles in vector for passing to gs setup
4221  std::vector< Ulong > handle_vec; // Assumes that we can do conversion from Ulong to EntityHandle
4222  std::copy( skin_ents[0].begin(), skin_ents[0].end(), std::back_inserter( handle_vec ) );
4223 
4224 #ifdef MOAB_HAVE_MPE
4225  if( myDebug->get_verbosity() == 2 )
4226  {
4227  MPE_Log_event( SHAREDV_START, procConfig.proc_rank(), "Creating crystal router." );
4228  }
4229 #endif
4230 
4231  // Get a crystal router
4232  gs_data::crystal_data* cd = procConfig.crystal_router();
4233 
4234  /*
4235  // Get total number of entities; will overshoot highest global id, but
4236  // that's OK
4237  int num_total[2] = {0, 0}, num_local[2] = {0, 0};
4238  result = mbImpl->get_number_entities_by_dimension(this_set, 0, num_local);
4239  if (MB_SUCCESS != result)return result;
4240  int failure = MPI_Allreduce(num_local, num_total, 1,
4241  MPI_INT, MPI_SUM, procConfig.proc_comm());
4242  if (failure) {
4243  MB_SET_ERR(MB_FAILURE, "Allreduce for total number of shared ents failed");
4244  }
4245  */
4246  // Call gather-scatter to get shared ids & procs
4247  gs_data* gsd = new gs_data();
4248  // assert(sizeof(ulong_) == sizeof(EntityHandle));
4249  result = gsd->initialize( skin_ents[0].size(), lgid_data.data(), handle_vec.data(), 2, 1, 1, cd );
4250  MB_CHK_SET_ERR( result, "Failed to create gs data" );
4251 
4252  // Get shared proc tags
4253  Tag shp_tag, shps_tag, shh_tag, shhs_tag, pstat_tag;
4254  result = get_shared_proc_tags( shp_tag, shps_tag, shh_tag, shhs_tag, pstat_tag );
4255  MB_CHK_SET_ERR( result, "Failed to get shared proc tags" );
4256 
4257  // Load shared verts into a tuple, then sort by index
4258  TupleList shared_verts;
4259  shared_verts.initialize( 2, 0, 1, 0, skin_ents[0].size() * ( MAX_SHARING_PROCS + 1 ) );
4260  shared_verts.enableWriteAccess();
4261 
4262  unsigned int i = 0, j = 0;
4263  for( unsigned int p = 0; p < gsd->nlinfo->_np; p++ )
4264  for( unsigned int np = 0; np < gsd->nlinfo->_nshared[p]; np++ )
4265  {
4266  shared_verts.vi_wr[i++] = gsd->nlinfo->_sh_ind[j];
4267  shared_verts.vi_wr[i++] = gsd->nlinfo->_target[p];
4268  shared_verts.vul_wr[j] = gsd->nlinfo->_ulabels[j];
4269  j++;
4270  shared_verts.inc_n();
4271  }
4272 
4273  myDebug->tprintf( 3, " shared verts size %d \n", (int)shared_verts.get_n() );
4274 
4275  int max_size = skin_ents[0].size() * ( MAX_SHARING_PROCS + 1 );
4276  moab::TupleList::buffer sort_buffer;
4277  sort_buffer.buffer_init( max_size );
4278  shared_verts.sort( 0, &sort_buffer );
4279  sort_buffer.reset();
4280 
4281  // Set sharing procs and handles tags on skin ents
4282  int maxp = -1;
4283  std::vector< int > sharing_procs( MAX_SHARING_PROCS );
4284  std::fill( sharing_procs.begin(), sharing_procs.end(), maxp );
4285  j = 0;
4286  i = 0;
4287 
4288  // Get ents shared by 1 or n procs
4289  std::map< std::vector< int >, std::vector< EntityHandle > > proc_nvecs;
4290  Range proc_verts;
4291  result = mbImpl->get_adjacencies( proc_ents, 0, false, proc_verts, Interface::UNION );
4292  MB_CHK_SET_ERR( result, "Failed to get proc_verts" );
4293 
4294  myDebug->print( 3, " resolve shared ents: proc verts ", proc_verts );
4295  result = tag_shared_verts( shared_verts, skin_ents, proc_nvecs, proc_verts );
4296  MB_CHK_SET_ERR( result, "Failed to tag shared verts" );
4297 
4298 #ifdef MOAB_HAVE_MPE
4299  if( myDebug->get_verbosity() == 2 )
4300  {
4301  MPE_Log_event( SHAREDV_END, procConfig.proc_rank(), "Finished tag_shared_verts." );
4302  }
4303 #endif
4304 
4305  // Get entities shared by 1 or n procs
4306  result = get_proc_nvecs( resolve_dim, shared_dim, skin_ents, proc_nvecs );
4307  MB_CHK_SET_ERR( result, "Failed to tag shared entities" );
4308 
4309  shared_verts.reset();
4310 
4311  if( myDebug->get_verbosity() > 0 )
4312  {
4313  for( std::map< std::vector< int >, std::vector< EntityHandle > >::const_iterator mit = proc_nvecs.begin();
4314  mit != proc_nvecs.end(); ++mit )
4315  {
4316  myDebug->tprintf( 1, "Iface: " );
4317  for( std::vector< int >::const_iterator vit = ( mit->first ).begin(); vit != ( mit->first ).end(); ++vit )
4318  myDebug->printf( 1, " %d", *vit );
4319  myDebug->print( 1, "\n" );
4320  }
4321  }
4322 
4323  // Create the sets for each interface; store them as tags on
4324  // the interface instance
4325  Range iface_sets;
4326  result = create_interface_sets( proc_nvecs );
4327  MB_CHK_SET_ERR( result, "Failed to create interface sets" );
4328 
4329  // Establish comm procs and buffers for them
4330  std::set< unsigned int > procs;
4331  result = get_interface_procs( procs, true );
4332  MB_CHK_SET_ERR( result, "Failed to get interface procs" );
4333 
4334 #ifndef NDEBUG
4335  result = check_all_shared_handles( true );
4336  MB_CHK_SET_ERR( result, "Shared handle check failed after interface vertex exchange" );
4337 #endif
4338 
4339  // Resolve shared entity remote handles; implemented in ghost cell exchange
4340  // code because it's so similar
4341  result = exchange_ghost_cells( -1, -1, 0, 0, true, true );
4342  MB_CHK_SET_ERR( result, "Failed to resolve shared entity remote handles" );
4343 
4344  // Now build parent/child links for interface sets
4345  result = create_iface_pc_links();
4346  MB_CHK_SET_ERR( result, "Failed to create interface parent/child links" );
4347 
4348  gsd->reset();
4349  delete gsd;
4350 
4351 #ifdef MOAB_HAVE_MPE
4352  if( myDebug->get_verbosity() == 2 )
4353  {
4354  MPE_Log_event( RESOLVE_END, procConfig.proc_rank(), "Exiting resolve_shared_ents." );
4355  }
4356 #endif
4357 
4358  // std::ostringstream ent_str;
4359  // ent_str << "mesh." << procConfig.proc_rank() << ".h5m";
4360  // mbImpl->write_mesh(ent_str.str().c_str());
4361 
4362  // Done
4363  return result;
4364 }

References moab::Interface::add_entities(), assign_global_ids(), moab::Range::begin(), check_all_shared_handles(), create_iface_pc_links(), create_interface_sets(), moab::ProcConfig::crystal_router(), define_mpe(), moab::Interface::dimension_from_handle(), moab::Range::empty(), moab::TupleList::enableWriteAccess(), ErrorCode, exchange_ghost_cells(), moab::Skinner::find_skin(), moab::Interface::get_adjacencies(), get_interface_procs(), moab::TupleList::get_n(), get_proc_nvecs(), get_shared_proc_tags(), moab::DebugOutput::get_verbosity(), GLOBAL_ID_TAG_NAME, moab::TupleList::inc_n(), moab::TupleList::initialize(), moab::gs_data::initialize(), MAX_SHARING_PROCS, MB_ALREADY_ALLOCATED, MB_CHK_ERR, MB_CHK_SET_ERR, MB_SET_ERR, MB_SUCCESS, MB_TAG_CREAT, MB_TAG_DENSE, MB_TYPE_HANDLE, MB_TYPE_INTEGER, MB_TYPE_OPAQUE, mbImpl, MPE_Log_event, myDebug, moab::DebugOutput::print(), moab::DebugOutput::printf(), proc_config(), moab::ProcConfig::proc_rank(), procConfig, moab::TupleList::buffer::reset(), moab::TupleList::reset(), moab::gs_data::reset(), moab::Range::size(), size(), moab::TupleList::sort(), moab::Interface::tag_get_bytes(), moab::Interface::tag_get_data(), moab::Interface::tag_get_data_type(), moab::Interface::tag_get_handle(), tag_shared_verts(), moab::DebugOutput::tprintf(), moab::Interface::UNION, moab::TupleList::vi_wr, and moab::TupleList::vul_wr.

Referenced by iMOAB_ReceiveMesh(), iMOAB_ResolveSharedEntities(), moab::ReadParallel::load_file(), and resolve_shared_ents().

◆ resolve_shared_ents() [3/3]

ErrorCode moab::ParallelComm::resolve_shared_ents ( ParallelComm **  pc,
const unsigned int  np,
EntityHandle  this_set,
const int  to_dim 
)
static

Definition at line 4398 of file ParallelComm.cpp.

4402 {
4403  std::vector< Range > verts( np );
4404  int tot_verts = 0;
4405  unsigned int p, i, j, v;
4406  ErrorCode rval;
4407  for( p = 0; p < np; p++ )
4408  {
4409  Skinner skinner( pc[p]->get_moab() );
4410  Range part_ents, skin_ents;
4411  rval = pc[p]->get_moab()->get_entities_by_dimension( this_set, part_dim, part_ents );
4412  if( MB_SUCCESS != rval ) return rval;
4413  rval = skinner.find_skin( this_set, part_ents, false, skin_ents, 0, true, true, true );
4414  if( MB_SUCCESS != rval ) return rval;
4415  rval = pc[p]->get_moab()->get_adjacencies( skin_ents, 0, true, verts[p], Interface::UNION );
4416  if( MB_SUCCESS != rval ) return rval;
4417  tot_verts += verts[p].size();
4418  }
4419 
4420  TupleList shared_ents;
4421  shared_ents.initialize( 2, 0, 1, 0, tot_verts );
4422  shared_ents.enableWriteAccess();
4423 
4424  i = 0;
4425  j = 0;
4426  std::vector< int > gids;
4427  Range::iterator rit;
4428  Tag gid_tag;
4429  for( p = 0; p < np; p++ )
4430  {
4431  gid_tag = pc[p]->get_moab()->globalId_tag();
4432 
4433  gids.resize( verts[p].size() );
4434  rval = pc[p]->get_moab()->tag_get_data( gid_tag, verts[p], gids.data() );
4435  if( MB_SUCCESS != rval ) return rval;
4436 
4437  for( v = 0, rit = verts[p].begin(); v < gids.size(); v++, ++rit )
4438  {
4439  shared_ents.vi_wr[i++] = gids[v];
4440  shared_ents.vi_wr[i++] = p;
4441  shared_ents.vul_wr[j] = *rit;
4442  j++;
4443  shared_ents.inc_n();
4444  }
4445  }
4446 
4447  moab::TupleList::buffer sort_buffer;
4448  sort_buffer.buffer_init( tot_verts );
4449  shared_ents.sort( 0, &sort_buffer );
4450  sort_buffer.reset();
4451 
4452  j = 0;
4453  i = 0;
4454  std::vector< EntityHandle > handles;
4455  std::vector< int > procs;
4456 
4457  while( i < shared_ents.get_n() )
4458  {
4459  handles.clear();
4460  procs.clear();
4461 
4462  // Count & accumulate sharing procs
4463  int this_gid = shared_ents.vi_rd[j];
4464  while( i < shared_ents.get_n() && shared_ents.vi_rd[j] == this_gid )
4465  {
4466  j++;
4467  procs.push_back( shared_ents.vi_rd[j++] );
4468  handles.push_back( shared_ents.vul_rd[i++] );
4469  }
4470  if( 1 == procs.size() ) continue;
4471 
4472  for( v = 0; v < procs.size(); v++ )
4473  {
4474  rval = pc[procs[v]]->update_remote_data( handles[v], procs.data(), handles.data(), procs.size(),
4475  ( procs[0] == (int)pc[procs[v]]->rank()
4476  ? PSTATUS_INTERFACE
4477  : ( PSTATUS_NOT_OWNED | PSTATUS_INTERFACE ) ) );
4478  if( MB_SUCCESS != rval ) return rval;
4479  }
4480  }
4481 
4482  std::set< unsigned int > psets;
4483  for( p = 0; p < np; p++ )
4484  {
4485  rval = pc[p]->create_interface_sets( this_set, part_dim, part_dim - 1 );
4486  if( MB_SUCCESS != rval ) return rval;
4487  // Establish comm procs and buffers for them
4488  psets.clear();
4489  rval = pc[p]->get_interface_procs( psets, true );
4490  if( MB_SUCCESS != rval ) return rval;
4491  }
4492 
4493  shared_ents.reset();
4494 
4495  return MB_SUCCESS;
4496 }

References create_interface_sets(), moab::TupleList::enableWriteAccess(), ErrorCode, moab::Skinner::find_skin(), moab::Interface::get_adjacencies(), moab::Interface::get_entities_by_dimension(), get_interface_procs(), get_moab(), moab::TupleList::get_n(), moab::Interface::globalId_tag(), moab::TupleList::inc_n(), moab::TupleList::initialize(), MB_SUCCESS, PSTATUS_INTERFACE, PSTATUS_NOT_OWNED, rank(), moab::TupleList::buffer::reset(), moab::TupleList::reset(), size(), moab::TupleList::sort(), moab::Interface::tag_get_data(), moab::Interface::UNION, update_remote_data(), moab::TupleList::vi_rd, moab::TupleList::vi_wr, moab::TupleList::vul_rd, and moab::TupleList::vul_wr.

◆ resolve_shared_sets() [1/2]

ErrorCode moab::ParallelComm::resolve_shared_sets ( EntityHandle  this_set,
const Tag *  id_tag = 0 
)

Remove shared sets.

Generates list of candidate sets using from those (directly) contained in passed set and passes them to the other version of resolve_shared_sets.

Parameters
this_setSet directly containing candidate sets (e.g. file set)
id_tagTag containing global IDs for entity sets.

Definition at line 4653 of file ParallelComm.cpp.

4654 {
4655  // Find all sets with any of the following tags:
4656  const char* const shared_set_tag_names[] = { GEOM_DIMENSION_TAG_NAME, MATERIAL_SET_TAG_NAME, DIRICHLET_SET_TAG_NAME,
4657  NEUMANN_SET_TAG_NAME, PARALLEL_PARTITION_TAG_NAME };
4658  int num_tags = sizeof( shared_set_tag_names ) / sizeof( shared_set_tag_names[0] );
4659  Range candidate_sets;
4660  ErrorCode result = MB_FAILURE;
4661 
4662  // If we're not given an ID tag to use to globally identify sets,
4663  // then fall back to using known tag values
4664  if( !idtag )
4665  {
4666  Tag gid, tag;
4667  gid = mbImpl->globalId_tag();
4668  if( NULL != gid ) result = mbImpl->tag_get_handle( GEOM_DIMENSION_TAG_NAME, 1, MB_TYPE_INTEGER, tag );
4669  if( MB_SUCCESS == result )
4670  {
4671  for( int d = 0; d < 4; d++ )
4672  {
4673  candidate_sets.clear();
4674  const void* vals[] = { &d };
4675  result = mbImpl->get_entities_by_type_and_tag( file, MBENTITYSET, &tag, vals, 1, candidate_sets );
4676  if( MB_SUCCESS == result ) resolve_shared_sets( candidate_sets, gid );
4677  }
4678  }
4679 
4680  for( int i = 1; i < num_tags; i++ )
4681  {
4682  result = mbImpl->tag_get_handle( shared_set_tag_names[i], 1, MB_TYPE_INTEGER, tag );
4683  if( MB_SUCCESS == result )
4684  {
4685  candidate_sets.clear();
4686  result = mbImpl->get_entities_by_type_and_tag( file, MBENTITYSET, &tag, 0, 1, candidate_sets );
4687  if( MB_SUCCESS == result ) resolve_shared_sets( candidate_sets, tag );
4688  }
4689  }
4690 
4691  return MB_SUCCESS;
4692  }
4693 
4694  for( int i = 0; i < num_tags; i++ )
4695  {
4696  Tag tag;
4697  result = mbImpl->tag_get_handle( shared_set_tag_names[i], 1, MB_TYPE_INTEGER, tag, MB_TAG_ANY );
4698  if( MB_SUCCESS != result ) continue;
4699 
4700  mbImpl->get_entities_by_type_and_tag( file, MBENTITYSET, &tag, 0, 1, candidate_sets, Interface::UNION );
4701  }
4702 
4703  // Find any additional sets that contain shared entities
4704  Range::iterator hint = candidate_sets.begin();
4705  Range all_sets;
4706  mbImpl->get_entities_by_type( file, MBENTITYSET, all_sets );
4707  all_sets = subtract( all_sets, candidate_sets );
4708  Range::iterator it = all_sets.begin();
4709  while( it != all_sets.end() )
4710  {
4711  Range contents;
4712  mbImpl->get_entities_by_handle( *it, contents );
4713  contents.erase( contents.lower_bound( MBENTITYSET ), contents.end() );
4714  filter_pstatus( contents, PSTATUS_SHARED, PSTATUS_OR );
4715  if( contents.empty() )
4716  {
4717  ++it;
4718  }
4719  else
4720  {
4721  hint = candidate_sets.insert( hint, *it );
4722  it = all_sets.erase( it );
4723  }
4724  }
4725 
4726  // Find any additionl sets that contain or are parents of potential shared sets
4727  Range prev_list = candidate_sets;
4728  while( !prev_list.empty() )
4729  {
4730  it = all_sets.begin();
4731  Range new_list;
4732  hint = new_list.begin();
4733  while( it != all_sets.end() )
4734  {
4735  Range contents;
4736  mbImpl->get_entities_by_type( *it, MBENTITYSET, contents );
4737  if( !intersect( prev_list, contents ).empty() )
4738  {
4739  hint = new_list.insert( hint, *it );
4740  it = all_sets.erase( it );
4741  }
4742  else
4743  {
4744  new_list.clear();
4745  mbImpl->get_child_meshsets( *it, contents );
4746  if( !intersect( prev_list, contents ).empty() )
4747  {
4748  hint = new_list.insert( hint, *it );
4749  it = all_sets.erase( it );
4750  }
4751  else
4752  {
4753  ++it;
4754  }
4755  }
4756  }
4757 
4758  candidate_sets.merge( new_list );
4759  prev_list.swap( new_list );
4760  }
4761 
4762  return resolve_shared_sets( candidate_sets, *idtag );
4763 }

References moab::Range::begin(), moab::Range::clear(), DIRICHLET_SET_TAG_NAME, moab::Range::empty(), moab::Range::end(), moab::Range::erase(), ErrorCode, filter_pstatus(), GEOM_DIMENSION_TAG_NAME, moab::Interface::get_child_meshsets(), moab::Interface::get_entities_by_handle(), moab::Interface::get_entities_by_type(), moab::Interface::get_entities_by_type_and_tag(), moab::Interface::globalId_tag(), moab::Range::insert(), moab::intersect(), moab::Range::lower_bound(), MATERIAL_SET_TAG_NAME, MB_SUCCESS, MB_TAG_ANY, MB_TYPE_INTEGER, MBENTITYSET, mbImpl, moab::Range::merge(), NEUMANN_SET_TAG_NAME, PARALLEL_PARTITION_TAG_NAME, PSTATUS_OR, PSTATUS_SHARED, moab::subtract(), moab::Range::swap(), moab::Interface::tag_get_handle(), and moab::Interface::UNION.

Referenced by moab::ReadParallel::load_file().

◆ resolve_shared_sets() [2/2]

ErrorCode moab::ParallelComm::resolve_shared_sets ( Range &  candidate_sets,
Tag  id_tag 
)

Remove shared sets.

Use values of id_tag to match sets across processes and populate sharing data for sets.

Parameters
candidate_setsSets to consider as potentially shared.
id_tagTag containing global IDs for entity sets.

Definition at line 4774 of file ParallelComm.cpp.

4775 {
4776  ErrorCode result;
4777  const unsigned rk = proc_config().proc_rank();
4778  MPI_Comm cm = proc_config().proc_comm();
4779 
4780  // Build sharing list for all sets
4781 
4782  // Get ids for sets in a vector, to pass to gs
4783  std::vector< long > larray; // Allocate sufficient space for longs
4784  std::vector< Ulong > handles;
4785  Range tmp_sets;
4786  // The id tag can be size 4 or size 8
4787  // Based on that, convert to int or to long, similarly to what we do
4788  // for resolving shared vertices;
4789  // This code must work on 32 bit too, where long is 4 bytes, also
4790  // so test first size 4, then we should be fine
4791  DataType tag_type;
4792  result = mbImpl->tag_get_data_type( idtag, tag_type );
4793  MB_CHK_SET_ERR( result, "Failed getting tag data type" );
4794  int bytes_per_tag;
4795  result = mbImpl->tag_get_bytes( idtag, bytes_per_tag );
4796  MB_CHK_SET_ERR( result, "Failed getting number of bytes per tag" );
4797  // On 64 bits, long and int are different
4798  // On 32 bits, they are not; if size of long is 8, it is a 64 bit machine (really?)
4799 
4800  for( Range::iterator rit = sets.begin(); rit != sets.end(); ++rit )
4801  {
4802  if( sizeof( long ) == bytes_per_tag && ( ( MB_TYPE_HANDLE == tag_type ) || ( MB_TYPE_OPAQUE == tag_type ) ) )
4803  { // It is a special id tag
4804  long dum;
4805  result = mbImpl->tag_get_data( idtag, &( *rit ), 1, &dum );
4806  if( MB_SUCCESS == result )
4807  {
4808  larray.push_back( dum );
4809  handles.push_back( *rit );
4810  tmp_sets.insert( tmp_sets.end(), *rit );
4811  }
4812  }
4813  else if( 4 == bytes_per_tag )
4814  { // Must be GLOBAL_ID tag or MATERIAL_ID, etc
4815  int dum;
4816  result = mbImpl->tag_get_data( idtag, &( *rit ), 1, &dum );
4817  if( MB_SUCCESS == result )
4818  {
4819  larray.push_back( dum );
4820  handles.push_back( *rit );
4821  tmp_sets.insert( tmp_sets.end(), *rit );
4822  }
4823  }
4824  }
4825 
4826  const size_t nsets = handles.size();
4827 
4828  // Get handle array for sets
4829  // This is not true on windows machine, 64 bits: entity handle is 64 bit, long is 32
4830  // assert(sizeof(EntityHandle) <= sizeof(unsigned long));
4831 
4832  // Do communication of data
4833  gs_data::crystal_data* cd = procConfig.crystal_router();
4834  gs_data* gsd = new gs_data();
4835  result = gsd->initialize( nsets, larray.data(), handles.data(), 2, 1, 1, cd );
4836  MB_CHK_SET_ERR( result, "Failed to create gs data" );
4837 
4838  // Convert from global IDs grouped by process rank to list
4839  // of <idx, rank> pairs so that we can sort primarily
4840  // by idx and secondarily by rank (we want lists of procs for each
4841  // idx, not lists if indices for each proc).
4842  size_t ntuple = 0;
4843  for( unsigned p = 0; p < gsd->nlinfo->_np; p++ )
4844  ntuple += gsd->nlinfo->_nshared[p];
4845  std::vector< set_tuple > tuples;
4846  tuples.reserve( ntuple );
4847  size_t j = 0;
4848  for( unsigned p = 0; p < gsd->nlinfo->_np; p++ )
4849  {
4850  for( unsigned np = 0; np < gsd->nlinfo->_nshared[p]; np++ )
4851  {
4852  set_tuple t;
4853  t.idx = gsd->nlinfo->_sh_ind[j];
4854  t.proc = gsd->nlinfo->_target[p];
4855  t.handle = gsd->nlinfo->_ulabels[j];
4856  tuples.push_back( t );
4857  j++;
4858  }
4859  }
4860  std::sort( tuples.begin(), tuples.end() );
4861 
4862  // Release crystal router stuff
4863  gsd->reset();
4864  delete gsd;
4865 
4866  // Storing sharing data for each set
4867  size_t ti = 0;
4868  unsigned idx = 0;
4869  std::vector< unsigned > procs;
4870  Range::iterator si = tmp_sets.begin();
4871  while( si != tmp_sets.end() && ti < tuples.size() )
4872  {
4873  assert( idx <= tuples[ti].idx );
4874  if( idx < tuples[ti].idx ) si += ( tuples[ti].idx - idx );
4875  idx = tuples[ti].idx;
4876 
4877  procs.clear();
4878  size_t ti_init = ti;
4879  while( ti < tuples.size() && tuples[ti].idx == idx )
4880  {
4881  procs.push_back( tuples[ti].proc );
4882  ++ti;
4883  }
4884  assert( is_sorted_unique( procs ) );
4885 
4886  result = sharedSetData->set_sharing_procs( *si, procs );
4887  if( MB_SUCCESS != result )
4888  {
4889  std::cerr << "Failure at " __FILE__ ":" << __LINE__ << std::endl;
4890  std::cerr.flush();
4891  MPI_Abort( cm, 1 );
4892  }
4893 
4894  // Add this proc to list of sharing procs in correct position
4895  // so that all procs select owner based on same list
4896  std::vector< unsigned >::iterator it = std::lower_bound( procs.begin(), procs.end(), rk );
4897  assert( it == procs.end() || *it > rk );
4898  procs.insert( it, rk );
4899  size_t owner_idx = choose_owner_idx( procs );
4900  EntityHandle owner_handle;
4901  if( procs[owner_idx] == rk )
4902  owner_handle = *si;
4903  else if( procs[owner_idx] > rk )
4904  owner_handle = tuples[ti_init + owner_idx - 1].handle;
4905  else
4906  owner_handle = tuples[ti_init + owner_idx].handle;
4907  result = sharedSetData->set_owner( *si, procs[owner_idx], owner_handle );
4908  if( MB_SUCCESS != result )
4909  {
4910  std::cerr << "Failure at " __FILE__ ":" << __LINE__ << std::endl;
4911  std::cerr.flush();
4912  MPI_Abort( cm, 1 );
4913  }
4914 
4915  ++si;
4916  ++idx;
4917  }
4918 
4919  return MB_SUCCESS;
4920 }

References moab::Range::begin(), moab::choose_owner_idx(), moab::ProcConfig::crystal_router(), moab::dum, moab::Range::end(), ErrorCode, moab::set_tuple::handle, moab::set_tuple::idx, moab::gs_data::initialize(), moab::Range::insert(), moab::is_sorted_unique(), MB_CHK_SET_ERR, MB_SUCCESS, MB_TYPE_HANDLE, MB_TYPE_OPAQUE, mbImpl, moab::set_tuple::proc, moab::ProcConfig::proc_comm(), proc_config(), moab::ProcConfig::proc_rank(), procConfig, moab::gs_data::reset(), moab::SharedSetData::set_owner(), moab::SharedSetData::set_sharing_procs(), sharedSetData, moab::Interface::tag_get_bytes(), moab::Interface::tag_get_data(), and moab::Interface::tag_get_data_type().

◆ scatter_entities()

ErrorCode moab::ParallelComm::scatter_entities ( const int  from_proc,
std::vector< Range > &  entities,
const bool  adjacencies = false,
const bool  tags = true 
)

Scatter entities on from_proc to other processors This function assumes remote handles are not being stored, since (usually) every processor will know about the whole mesh.

Parameters
from_procProcessor having the mesh to be broadcast
entitiesOn return, the entities sent or received in this call
adjacenciesIf true, adjacencies are sent for equiv entities (currently unsupported)
tagsIf true, all non-default-valued tags are sent for sent entities

Definition at line 607 of file ParallelComm.cpp.

611 {
612 #ifndef MOAB_HAVE_MPI
613  return MB_FAILURE;
614 #else
615  ErrorCode result = MB_SUCCESS;
616  int i, success, buff_size, prev_size;
617  int nProcs = (int)procConfig.proc_size();
618  int* sendCounts = new int[nProcs];
619  int* displacements = new int[nProcs];
620  sendCounts[0] = sizeof( int );
621  displacements[0] = 0;
622  Buffer buff( INITIAL_BUFF_SIZE );
623  buff.reset_ptr( sizeof( int ) );
624  buff.set_stored_size();
625  unsigned int my_proc = procConfig.proc_rank();
626 
627  // Get buffer size array for each remote processor
628  if( my_proc == (unsigned int)from_proc )
629  {
630  for( i = 1; i < nProcs; i++ )
631  {
632  prev_size = buff.buff_ptr - buff.mem_ptr;
633  buff.reset_ptr( prev_size + sizeof( int ) );
634  result = add_verts( entities[i] );
635  MB_CHK_SET_ERR( result, "Failed to add verts" );
636 
637  result = pack_buffer( entities[i], adjacencies, tags, false, -1, &buff );
638  if( MB_SUCCESS != result )
639  {
640  delete[] sendCounts;
641  delete[] displacements;
642  MB_SET_ERR( result, "Failed to pack buffer in scatter_entities" );
643  }
644 
645  buff_size = buff.buff_ptr - buff.mem_ptr - prev_size;
646  *( (int*)( buff.mem_ptr + prev_size ) ) = buff_size;
647  sendCounts[i] = buff_size;
648  }
649  }
650 
651  // Broadcast buffer size array
652  success = MPI_Bcast( sendCounts, nProcs, MPI_INT, from_proc, procConfig.proc_comm() );
653  if( MPI_SUCCESS != success )
654  {
655  delete[] sendCounts;
656  delete[] displacements;
657  MB_SET_ERR( MB_FAILURE, "MPI_Bcast of buffer size failed" );
658  }
659 
660  for( i = 1; i < nProcs; i++ )
661  {
662  displacements[i] = displacements[i - 1] + sendCounts[i - 1];
663  }
664 
665  Buffer rec_buff;
666  rec_buff.reserve( sendCounts[my_proc] );
667 
668  // Scatter actual geometry
669  success = MPI_Scatterv( buff.mem_ptr, sendCounts, displacements, MPI_UNSIGNED_CHAR, rec_buff.mem_ptr,
670  sendCounts[my_proc], MPI_UNSIGNED_CHAR, from_proc, procConfig.proc_comm() );
671 
672  if( MPI_SUCCESS != success )
673  {
674  delete[] sendCounts;
675  delete[] displacements;
676  MB_SET_ERR( MB_FAILURE, "MPI_Scatterv of buffer failed" );
677  }
678 
679  // Unpack in remote processors
680  if( my_proc != (unsigned int)from_proc )
681  {
682  std::vector< std::vector< EntityHandle > > dum1a, dum1b;
683  std::vector< std::vector< int > > dum1p;
684  std::vector< EntityHandle > dum2, dum4;
685  std::vector< unsigned int > dum3;
686  rec_buff.reset_ptr( sizeof( int ) );
687  result = unpack_buffer( rec_buff.buff_ptr, false, from_proc, -1, dum1a, dum1b, dum1p, dum2, dum2, dum3, dum4 );
688  if( MB_SUCCESS != result )
689  {
690  delete[] sendCounts;
691  delete[] displacements;
692  MB_SET_ERR( result, "Failed to unpack buffer in scatter_entities" );
693  }
694 
695  std::copy( dum4.begin(), dum4.end(), range_inserter( entities[my_proc] ) );
696  }
697 
698  delete[] sendCounts;
699  delete[] displacements;
700 
701  return MB_SUCCESS;
702 #endif
703 }

References add_verts(), moab::ParallelComm::Buffer::buff_ptr, ErrorCode, INITIAL_BUFF_SIZE, MB_CHK_SET_ERR, MB_SET_ERR, MB_SUCCESS, moab::ParallelComm::Buffer::mem_ptr, pack_buffer(), moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_rank(), moab::ProcConfig::proc_size(), procConfig, moab::ParallelComm::Buffer::reserve(), moab::ParallelComm::Buffer::reset_ptr(), moab::ParallelComm::Buffer::set_stored_size(), and unpack_buffer().

◆ send_buffer()

ErrorCode moab::ParallelComm::send_buffer ( const unsigned int  to_proc,
Buffer *  send_buff,
const int  msg_tag,
MPI_Request &  send_req,
MPI_Request &  ack_recv_req,
int *  ack_buff,
int &  this_incoming,
int  next_mesg_tag = -1,
Buffer *  next_recv_buff = NULL,
MPI_Request *  next_recv_req = NULL,
int *  next_incoming = NULL 
)
private

send the indicated buffer, possibly sending size first

Definition at line 6303 of file ParallelComm.cpp.

6314 {
6315  ErrorCode result = MB_SUCCESS;
6316  int success;
6317 
6318  // If small message, post recv for remote handle message
6319  if( send_buff->get_stored_size() <= (int)INITIAL_BUFF_SIZE && next_recv_buff )
6320  {
6321  ( *next_incoming )++;
6322  PRINT_DEBUG_IRECV( procConfig.proc_rank(), to_proc, next_recv_buff->mem_ptr, INITIAL_BUFF_SIZE, next_mesg_tag,
6323  *next_incoming );
6324  success = MPI_Irecv( next_recv_buff->mem_ptr, INITIAL_BUFF_SIZE, MPI_UNSIGNED_CHAR, to_proc, next_mesg_tag,
6325  procConfig.proc_comm(), next_recv_req );
6326  if( success != MPI_SUCCESS )
6327  {
6328  MB_SET_ERR( MB_FAILURE, "Failed to post irecv for next message in ghost exchange" );
6329  }
6330  }
6331  // If large, we'll need an ack before sending the rest
6332  else if( send_buff->get_stored_size() > (int)INITIAL_BUFF_SIZE )
6333  {
6334  this_incoming++;
6335  PRINT_DEBUG_IRECV( procConfig.proc_rank(), to_proc, (unsigned char*)ack_buff, sizeof( int ), mesg_tag - 1,
6336  this_incoming );
6337  success = MPI_Irecv( (void*)ack_buff, sizeof( int ), MPI_UNSIGNED_CHAR, to_proc, mesg_tag - 1,
6338  procConfig.proc_comm(), &ack_req );
6339  if( success != MPI_SUCCESS )
6340  {
6341  MB_SET_ERR( MB_FAILURE, "Failed to post irecv for entity ack in ghost exchange" );
6342  }
6343  }
6344 
6345  // Send the buffer
6346  PRINT_DEBUG_ISEND( procConfig.proc_rank(), to_proc, send_buff->mem_ptr, mesg_tag,
6347  std::min( send_buff->get_stored_size(), (int)INITIAL_BUFF_SIZE ) );
6348  assert( 0 <= send_buff->get_stored_size() && send_buff->get_stored_size() <= (int)send_buff->alloc_size );
6349  success = MPI_Isend( send_buff->mem_ptr, std::min( send_buff->get_stored_size(), (int)INITIAL_BUFF_SIZE ),
6350  MPI_UNSIGNED_CHAR, to_proc, mesg_tag, procConfig.proc_comm(), &send_req );
6351  if( success != MPI_SUCCESS ) return MB_FAILURE;
6352 
6353  return result;
6354 }

References moab::ParallelComm::Buffer::alloc_size, ErrorCode, moab::ParallelComm::Buffer::get_stored_size(), INITIAL_BUFF_SIZE, MB_SET_ERR, MB_SUCCESS, moab::ParallelComm::Buffer::mem_ptr, PRINT_DEBUG_IRECV, PRINT_DEBUG_ISEND, moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_rank(), and procConfig.

Referenced by exchange_ghost_cells(), exchange_owned_mesh(), exchange_tags(), recv_entities(), recv_messages(), reduce_tags(), send_entities(), send_recv_entities(), and settle_intersection_points().

◆ send_entities() [1/2]

ErrorCode moab::ParallelComm::send_entities ( const int  to_proc,
Range &  orig_ents,
const bool  adjs,
const bool  tags,
const bool  store_remote_handles,
const bool  is_iface,
Range &  final_ents,
int &  incoming1,
int &  incoming2,
TupleList &  entprocs,
std::vector< MPI_Request > &  recv_remoteh_reqs,
bool  wait_all = true 
)

send entities to another processor, optionally waiting until it's done

Send entities to another processor, with adjs, sets, and tags. If store_remote_handles is true, this call receives back handles assigned to entities sent to destination processor and stores them in sharedh_tag or sharedhs_tag.

Parameters
to_procDestination processor
orig_entsEntities requested to send
adjsIf true, send adjacencies for equiv entities (currently unsupported)
tagsIf true, send tag values for all tags assigned to entities
store_remote_handlesIf true, also recv message with handles on destination processor (currently unsupported)
final_entsRange containing all entities sent
incomingkeep track if any messages are coming to this processor (newly added)
wait_allIf true, wait until all messages received/sent complete

Definition at line 705 of file ParallelComm.cpp.

717 {
718 #ifndef MOAB_HAVE_MPI
719  return MB_FAILURE;
720 #else
721  // Pack entities to local buffer
722  int ind = get_buffers( to_proc );
723  localOwnedBuffs[ind]->reset_ptr( sizeof( int ) );
724 
725  // Add vertices
726  ErrorCode result = add_verts( orig_ents );
727  MB_CHK_SET_ERR( result, "Failed to add verts in send_entities" );
728 
729  // Filter out entities already shared with destination
730  Range tmp_range;
731  result = filter_pstatus( orig_ents, PSTATUS_SHARED, PSTATUS_AND, to_proc, &tmp_range );
732  MB_CHK_SET_ERR( result, "Failed to filter on owner" );
733  if( !tmp_range.empty() )
734  {
735  orig_ents = subtract( orig_ents, tmp_range );
736  }
737 
738  result = pack_buffer( orig_ents, adjs, tags, store_remote_handles, to_proc, localOwnedBuffs[ind], &entprocs );
739  MB_CHK_SET_ERR( result, "Failed to pack buffer in send_entities" );
740 
741  // Send buffer
742  result = send_buffer( to_proc, localOwnedBuffs[ind], MB_MESG_ENTS_SIZE, sendReqs[2 * ind], recvReqs[2 * ind + 1],
743  (int*)( remoteOwnedBuffs[ind]->mem_ptr ),
744  //&ackbuff,
745  incoming1, MB_MESG_REMOTEH_SIZE,
746  ( !is_iface && store_remote_handles ? localOwnedBuffs[ind] : NULL ),
747  &recv_remoteh_reqs[2 * ind], &incoming2 );
748  MB_CHK_SET_ERR( result, "Failed to send buffer" );
749 
750  return MB_SUCCESS;
751 #endif
752 }

References add_verts(), moab::Range::empty(), ErrorCode, filter_pstatus(), get_buffers(), localOwnedBuffs, MB_CHK_SET_ERR, moab::MB_MESG_ENTS_SIZE, moab::MB_MESG_REMOTEH_SIZE, MB_SUCCESS, pack_buffer(), PSTATUS_AND, PSTATUS_SHARED, recvReqs, remoteOwnedBuffs, send_buffer(), sendReqs, and moab::subtract().

◆ send_entities() [2/2]

ErrorCode moab::ParallelComm::send_entities ( std::vector< unsigned int > &  send_procs,
std::vector< Range * > &  send_ents,
int &  incoming1,
int &  incoming2,
const bool  store_remote_handles 
)

Definition at line 754 of file ParallelComm.cpp.

759 {
760 #ifdef MOAB_HAVE_MPE
761  if( myDebug->get_verbosity() == 2 )
762  {
763  MPE_Log_event( OWNED_START, procConfig.proc_rank(), "Starting send_entities." );
764  }
765 #endif
766  myDebug->tprintf( 1, "Entering send_entities\n" );
767  if( myDebug->get_verbosity() == 4 )
768  {
769  msgs.clear();
770  msgs.reserve( MAX_SHARING_PROCS );
771  }
772 
773  unsigned int i;
774  int ind;
775  ErrorCode result = MB_SUCCESS;
776 
777  // Set buffProcs with communicating procs
778  unsigned int n_proc = send_procs.size();
779  for( i = 0; i < n_proc; i++ )
780  {
781  ind = get_buffers( send_procs[i] );
782  result = add_verts( *send_ents[i] );
783  MB_CHK_SET_ERR( result, "Failed to add verts" );
784 
785  // Filter out entities already shared with destination
786  Range tmp_range;
787  result = filter_pstatus( *send_ents[i], PSTATUS_SHARED, PSTATUS_AND, buffProcs[ind], &tmp_range );
788  MB_CHK_SET_ERR( result, "Failed to filter on owner" );
789  if( !tmp_range.empty() )
790  {
791  *send_ents[i] = subtract( *send_ents[i], tmp_range );
792  }
793  }
794 
795  //===========================================
796  // Get entities to be sent to neighbors
797  // Need to get procs each entity is sent to
798  //===========================================
799  Range allsent, tmp_range;
800  int npairs = 0;
801  TupleList entprocs;
802  for( i = 0; i < n_proc; i++ )
803  {
804  int n_ents = send_ents[i]->size();
805  if( n_ents > 0 )
806  {
807  npairs += n_ents; // Get the total # of proc/handle pairs
808  allsent.merge( *send_ents[i] );
809  }
810  }
811 
812  // Allocate a TupleList of that size
813  entprocs.initialize( 1, 0, 1, 0, npairs );
814  entprocs.enableWriteAccess();
815 
816  // Put the proc/handle pairs in the list
817  for( i = 0; i < n_proc; i++ )
818  {
819  for( Range::iterator rit = send_ents[i]->begin(); rit != send_ents[i]->end(); ++rit )
820  {
821  entprocs.vi_wr[entprocs.get_n()] = send_procs[i];
822  entprocs.vul_wr[entprocs.get_n()] = *rit;
823  entprocs.inc_n();
824  }
825  }
826 
827  // Sort by handle
828  moab::TupleList::buffer sort_buffer;
829  sort_buffer.buffer_init( npairs );
830  entprocs.sort( 1, &sort_buffer );
831  entprocs.disableWriteAccess();
832  sort_buffer.reset();
833 
834  myDebug->tprintf( 1, "allsent ents compactness (size) = %f (%lu)\n", allsent.compactness(),
835  (unsigned long)allsent.size() );
836 
837  //===========================================
838  // Pack and send ents from this proc to others
839  //===========================================
840  for( i = 0; i < n_proc; i++ )
841  {
842  if( send_ents[i]->size() > 0 )
843  {
844  ind = get_buffers( send_procs[i] );
845  myDebug->tprintf( 1, "Sent ents compactness (size) = %f (%lu)\n", send_ents[i]->compactness(),
846  (unsigned long)send_ents[i]->size() );
847  // Reserve space on front for size and for initial buff size
848  localOwnedBuffs[ind]->reset_buffer( sizeof( int ) );
849  result = pack_buffer( *send_ents[i], false, true, store_remote_handles, buffProcs[ind],
850  localOwnedBuffs[ind], &entprocs, &allsent );
851 
852  if( myDebug->get_verbosity() == 4 )
853  {
854  msgs.resize( msgs.size() + 1 );
855  msgs.back() = new Buffer( *localOwnedBuffs[ind] );
856  }
857 
858  // Send the buffer (size stored in front in send_buffer)
859  result = send_buffer( send_procs[i], localOwnedBuffs[ind], MB_MESG_ENTS_SIZE, sendReqs[2 * ind],
860  recvReqs[2 * ind + 1], &ackbuff, incoming1, MB_MESG_REMOTEH_SIZE,
861  ( store_remote_handles ? localOwnedBuffs[ind] : NULL ), &recvRemotehReqs[2 * ind],
862  &incoming2 );
863  MB_CHK_SET_ERR( result, "Failed to Isend in ghost send" );
864  }
865  }
866  entprocs.reset();
867 
868 #ifdef MOAB_HAVE_MPE
869  if( myDebug->get_verbosity() == 2 )
870  {
871  MPE_Log_event( ENTITIES_END, procConfig.proc_rank(), "Ending send_entities." );
872  }
873 #endif
874 
875  return MB_SUCCESS;
876 }

References ackbuff, add_verts(), buffProcs, moab::Range::compactness(), moab::TupleList::disableWriteAccess(), moab::Range::empty(), moab::TupleList::enableWriteAccess(), ErrorCode, filter_pstatus(), get_buffers(), moab::TupleList::get_n(), moab::DebugOutput::get_verbosity(), moab::TupleList::inc_n(), moab::TupleList::initialize(), localOwnedBuffs, MAX_SHARING_PROCS, MB_CHK_SET_ERR, moab::MB_MESG_ENTS_SIZE, moab::MB_MESG_REMOTEH_SIZE, MB_SUCCESS, moab::Range::merge(), MPE_Log_event, moab::msgs, myDebug, pack_buffer(), moab::ProcConfig::proc_rank(), procConfig, PSTATUS_AND, PSTATUS_SHARED, recvRemotehReqs, recvReqs, moab::TupleList::buffer::reset(), moab::TupleList::reset(), send_buffer(), sendReqs, moab::Range::size(), size(), moab::TupleList::sort(), moab::subtract(), moab::DebugOutput::tprintf(), moab::TupleList::vi_wr, and moab::TupleList::vul_wr.

◆ send_recv_entities()

ErrorCode moab::ParallelComm::send_recv_entities ( std::vector< int > &  send_procs,
std::vector< std::vector< int > > &  msgsizes,
std::vector< std::vector< EntityHandle > > &  senddata,
std::vector< std::vector< EntityHandle > > &  recvdata 
)

Send and receives data from a set of processors.

Definition at line 887 of file ParallelComm.cpp.

891 {
892 #ifdef USE_MPE
893  if( myDebug->get_verbosity() == 2 )
894  {
895  MPE_Log_event( OWNED_START, procConfig.proc_rank(), "Starting send_recv_entities." );
896  }
897 #endif
898  myDebug->tprintf( 1, "Entering send_recv_entities\n" );
899  if( myDebug->get_verbosity() == 4 )
900  {
901  msgs.clear();
902  msgs.reserve( MAX_SHARING_PROCS );
903  }
904 
905  // unsigned int i;
906  int i, ind, success;
907  ErrorCode error = MB_SUCCESS;
908 
909  //===========================================
910  // Pack and send ents from this proc to others
911  //===========================================
912 
913  // std::cout<<"resetting all buffers"<<std::endl;
914 
915  reset_all_buffers();
916  sendReqs.resize( 3 * buffProcs.size(), MPI_REQUEST_NULL );
917  std::vector< MPI_Request > recv_ent_reqs( 3 * buffProcs.size(), MPI_REQUEST_NULL );
918  int ack_buff;
919  int incoming = 0;
920 
921  std::vector< unsigned int >::iterator sit;
922 
923  for( ind = 0, sit = buffProcs.begin(); sit != buffProcs.end(); ++sit, ind++ )
924  {
925  incoming++;
926  PRINT_DEBUG_IRECV( *sit, procConfig.proc_rank(), remoteOwnedBuffs[ind]->mem_ptr, INITIAL_BUFF_SIZE,
927  MB_MESG_ENTS_SIZE, incoming );
928 
929  success = MPI_Irecv( remoteOwnedBuffs[ind]->mem_ptr, INITIAL_BUFF_SIZE, MPI_UNSIGNED_CHAR, *sit,
930  MB_MESG_ENTS_SIZE, procConfig.proc_comm(), &recv_ent_reqs[3 * ind] );
931  if( success != MPI_SUCCESS )
932  {
933  MB_SET_ERR( MB_FAILURE, "Failed to post irecv in send_recv_entities" );
934  }
935  }
936 
937  // std::set<unsigned int>::iterator it;
938  for( i = 0; i < (int)send_procs.size(); i++ )
939  {
940  // Get index of the shared processor in the local buffer
941  ind = get_buffers( send_procs[i] );
942  localOwnedBuffs[ind]->reset_buffer( sizeof( int ) );
943 
944  int buff_size = msgsizes[i].size() * sizeof( int ) + senddata[i].size() * sizeof( EntityHandle );
945  localOwnedBuffs[ind]->check_space( buff_size );
946 
947  // Pack entities
948  std::vector< int > msg;
949  msg.insert( msg.end(), msgsizes[i].begin(), msgsizes[i].end() );
950  PACK_INTS( localOwnedBuffs[ind]->buff_ptr, msg.data(), msg.size() );
951 
952  std::vector< EntityHandle > entities;
953  entities.insert( entities.end(), senddata[i].begin(), senddata[i].end() );
954  PACK_EH( localOwnedBuffs[ind]->buff_ptr, entities.data(), entities.size() );
955  localOwnedBuffs[ind]->set_stored_size();
956 
957  if( myDebug->get_verbosity() == 4 )
958  {
959  msgs.resize( msgs.size() + 1 );
960  msgs.back() = new Buffer( *localOwnedBuffs[ind] );
961  }
962 
963  // Send the buffer (size stored in front in send_buffer)
964  error = send_buffer( send_procs[i], localOwnedBuffs[ind], MB_MESG_ENTS_SIZE, sendReqs[3 * ind],
965  recv_ent_reqs[3 * ind + 2], &ack_buff, incoming );
966  MB_CHK_SET_ERR( error, "Failed to Isend in send_recv_entities" );
967  }
968 
969  //===========================================
970  // Receive and unpack ents from received data
971  //===========================================
972 
973  while( incoming )
974  {
975 
976  MPI_Status status;
977  int index_in_recv_requests;
978 
979  PRINT_DEBUG_WAITANY( recv_ent_reqs, MB_MESG_ENTS_SIZE, procConfig.proc_rank() );
980  success = MPI_Waitany( 3 * buffProcs.size(), recv_ent_reqs.data(), &index_in_recv_requests, &status );
981  if( MPI_SUCCESS != success )
982  {
983  MB_SET_ERR( MB_FAILURE, "Failed in waitany in send_recv_entities" );
984  }
985 
986  // Processor index in the list is divided by 3
987  ind = index_in_recv_requests / 3;
988 
989  PRINT_DEBUG_RECD( status );
990 
991  // OK, received something; decrement incoming counter
992  incoming--;
993 
994  bool done = false;
995 
996  error = recv_buffer( MB_MESG_ENTS_SIZE, status, remoteOwnedBuffs[ind],
997  recv_ent_reqs[3 * ind + 1], // This is for receiving the second message
998  recv_ent_reqs[3 * ind + 2], // This would be for ack, but it is not
999  // used; consider removing it
1000  incoming, localOwnedBuffs[ind],
1001  sendReqs[3 * ind + 1], // Send request for sending the second message
1002  sendReqs[3 * ind + 2], // This is for sending the ack
1003  done );
1004  MB_CHK_SET_ERR( error, "Failed to resize recv buffer" );
1005 
1006  if( done )
1007  {
1008  remoteOwnedBuffs[ind]->reset_ptr( sizeof( int ) );
1009 
1010  int from_proc = status.MPI_SOURCE;
1011  int idx = std::find( send_procs.begin(), send_procs.end(), from_proc ) - send_procs.begin();
1012 
1013  int msg = msgsizes[idx].size();
1014  std::vector< int > recvmsg( msg );
1015  int ndata = senddata[idx].size();
1016  std::vector< EntityHandle > dum_vec( ndata );
1017 
1018  UNPACK_INTS( remoteOwnedBuffs[ind]->buff_ptr, recvmsg.data(), msg );
1019  UNPACK_EH( remoteOwnedBuffs[ind]->buff_ptr, dum_vec.data(), ndata );
1020 
1021  recvdata[idx].insert( recvdata[idx].end(), dum_vec.begin(), dum_vec.end() );
1022  }
1023  }
1024 
1025 #ifdef USE_MPE
1026  if( myDebug->get_verbosity() == 2 )
1027  {
1028  MPE_Log_event( ENTITIES_END, procConfig.proc_rank(), "Ending send_recv_entities." );
1029  }
1030 #endif
1031 
1032  return MB_SUCCESS;
1033 }

References buffProcs, moab::error(), ErrorCode, get_buffers(), moab::DebugOutput::get_verbosity(), INITIAL_BUFF_SIZE, localOwnedBuffs, MAX_SHARING_PROCS, MB_CHK_SET_ERR, moab::MB_MESG_ENTS_SIZE, MB_SET_ERR, MB_SUCCESS, MPE_Log_event, moab::msgs, myDebug, moab::PACK_EH(), moab::PACK_INTS(), PRINT_DEBUG_IRECV, PRINT_DEBUG_RECD, PRINT_DEBUG_WAITANY, moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_rank(), procConfig, recv_buffer(), remoteOwnedBuffs, reset_all_buffers(), send_buffer(), sendReqs, size(), moab::DebugOutput::tprintf(), moab::UNPACK_EH(), and moab::UNPACK_INTS().

◆ set_debug_verbosity()

void moab::ParallelComm::set_debug_verbosity ( int  verb)

set the verbosity level of output from this pcomm

Definition at line 9201 of file ParallelComm.cpp.

9202 {
9203  myDebug->set_verbosity( verb );
9204 }

References myDebug, and moab::DebugOutput::set_verbosity().

Referenced by moab::ReadParallel::load_file().

◆ set_partitioning()

ErrorCode moab::ParallelComm::set_partitioning ( EntityHandle  h)

Definition at line 8394 of file ParallelComm.cpp.

8395 {
8396  ErrorCode rval;
8397  Tag prtn_tag;
8398  rval = mbImpl->tag_get_handle( PARTITIONING_PCOMM_TAG_NAME, 1, MB_TYPE_INTEGER, prtn_tag,
8399  MB_TAG_SPARSE | MB_TAG_CREAT );
8400  if( MB_SUCCESS != rval ) return rval;
8401 
8402  // Get my id
8403  ParallelComm* pcomm_arr[MAX_SHARING_PROCS];
8404  Tag pc_tag = pcomm_tag( mbImpl, false );
8405  if( 0 == pc_tag ) return MB_FAILURE;
8406  const EntityHandle root = 0;
8407  ErrorCode result = mbImpl->tag_get_data( pc_tag, &root, 1, pcomm_arr );
8408  if( MB_SUCCESS != result ) return MB_FAILURE;
8409  int id = std::find( pcomm_arr, pcomm_arr + MAX_SHARING_PROCS, this ) - pcomm_arr;
8410  if( id == MAX_SHARING_PROCS ) return MB_FAILURE;
8411 
8412  EntityHandle old = partitioningSet;
8413  if( old )
8414  {
8415  rval = mbImpl->tag_delete_data( prtn_tag, &old, 1 );
8416  if( MB_SUCCESS != rval ) return rval;
8417  partitioningSet = 0;
8418  }
8419 
8420  if( !set ) return MB_SUCCESS;
8421 
8422  Range contents;
8423  if( old )
8424  {
8425  rval = mbImpl->get_entities_by_handle( old, contents );
8426  if( MB_SUCCESS != rval ) return rval;
8427  }
8428  else
8429  {
8430  contents = partition_sets();
8431  }
8432 
8433  rval = mbImpl->add_entities( set, contents );
8434  if( MB_SUCCESS != rval ) return rval;
8435 
8436  // Store pcomm id on new partition set
8437  rval = mbImpl->tag_set_data( prtn_tag, &set, 1, &id );
8438  if( MB_SUCCESS != rval ) return rval;
8439 
8440  partitioningSet = set;
8441  return MB_SUCCESS;
8442 }

References moab::Interface::add_entities(), ErrorCode, moab::Interface::get_entities_by_handle(), MAX_SHARING_PROCS, MB_SUCCESS, MB_TAG_CREAT, MB_TAG_SPARSE, MB_TYPE_INTEGER, mbImpl, partition_sets(), moab::PARTITIONING_PCOMM_TAG_NAME, partitioningSet, pcomm_tag(), moab::Interface::tag_delete_data(), moab::Interface::tag_get_data(), moab::Interface::tag_get_handle(), and moab::Interface::tag_set_data().

Referenced by get_pcomm().

◆ set_pstatus_entities() [1/2]

ErrorCode moab::ParallelComm::set_pstatus_entities ( EntityHandle *  pstatus_ents,
int  num_ents,
unsigned char  pstatus_val,
bool  lower_dim_ents = false,
bool  verts_too = true,
int  operation = Interface::UNION 
)
private

Set pstatus values on entities (vector-based function)

Parameters
pstatus_entsEntities to be set
pstatus_valPstatus value to be set
lower_dim_entsIf true, lower-dimensional ents (incl. vertices) set too (and created if they don't exist)
verts_tooIf true, vertices also set
operationIf UNION, pstatus_val is OR-d with existing value, otherwise existing value is over-written

Definition at line 4596 of file ParallelComm.cpp.

4602 {
4603  std::vector< unsigned char > pstatus_vals( num_ents );
4604  ErrorCode result;
4605  if( lower_dim_ents || verts_too )
4606  {
4607  // In this case, call the range-based version
4608  Range tmp_range;
4609  std::copy( pstatus_ents, pstatus_ents + num_ents, range_inserter( tmp_range ) );
4610  return set_pstatus_entities( tmp_range, pstatus_val, lower_dim_ents, verts_too, operation );
4611  }
4612 
4613  if( Interface::UNION == operation )
4614  {
4615  result = mbImpl->tag_get_data( pstatus_tag(), pstatus_ents, num_ents, pstatus_vals.data() );
4616  MB_CHK_SET_ERR( result, "Failed to get pstatus tag data" );
4617  for( unsigned int i = 0; i < (unsigned int)num_ents; i++ )
4618  pstatus_vals[i] |= pstatus_val;
4619  }
4620  else
4621  {
4622  for( unsigned int i = 0; i < (unsigned int)num_ents; i++ )
4623  pstatus_vals[i] = pstatus_val;
4624  }
4625  result = mbImpl->tag_set_data( pstatus_tag(), pstatus_ents, num_ents, pstatus_vals.data() );
4626  MB_CHK_SET_ERR( result, "Failed to set pstatus tag data" );
4627 
4628  return MB_SUCCESS;
4629 }

References ErrorCode, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, pstatus_tag(), set_pstatus_entities(), moab::Interface::tag_get_data(), moab::Interface::tag_set_data(), and moab::Interface::UNION.

◆ set_pstatus_entities() [2/2]

ErrorCode moab::ParallelComm::set_pstatus_entities ( Range &  pstatus_ents,
unsigned char  pstatus_val,
bool  lower_dim_ents = false,
bool  verts_too = true,
int  operation = Interface::UNION 
)
private

Set pstatus values on entities.

Parameters
pstatus_entsEntities to be set
pstatus_valPstatus value to be set
lower_dim_entsIf true, lower-dimensional ents (incl. vertices) set too (and created if they don't exist)
verts_tooIf true, vertices also set
operationIf UNION, pstatus_val is OR-d with existing value, otherwise existing value is over-written

Definition at line 4558 of file ParallelComm.cpp.

4563 {
4564  std::vector< unsigned char > pstatus_vals( pstatus_ents.size() );
4565  Range all_ents, *range_ptr = &pstatus_ents;
4566  ErrorCode result;
4567  if( lower_dim_ents || verts_too )
4568  {
4569  all_ents = pstatus_ents;
4570  range_ptr = &all_ents;
4571  int start_dim = ( lower_dim_ents ? mbImpl->dimension_from_handle( *pstatus_ents.rbegin() ) - 1 : 0 );
4572  for( ; start_dim >= 0; start_dim-- )
4573  {
4574  result = mbImpl->get_adjacencies( all_ents, start_dim, true, all_ents, Interface::UNION );
4575  MB_CHK_SET_ERR( result, "Failed to get adjacencies for pstatus entities" );
4576  }
4577  }
4578  if( Interface::UNION == operation )
4579  {
4580  result = mbImpl->tag_get_data( pstatus_tag(), *range_ptr, pstatus_vals.data() );
4581  MB_CHK_SET_ERR( result, "Failed to get pstatus tag data" );
4582  for( unsigned int i = 0; i < pstatus_vals.size(); i++ )
4583  pstatus_vals[i] |= pstatus_val;
4584  }
4585  else
4586  {
4587  for( unsigned int i = 0; i < pstatus_vals.size(); i++ )
4588  pstatus_vals[i] = pstatus_val;
4589  }
4590  result = mbImpl->tag_set_data( pstatus_tag(), *range_ptr, pstatus_vals.data() );
4591  MB_CHK_SET_ERR( result, "Failed to set pstatus tag data" );
4592 
4593  return MB_SUCCESS;
4594 }

References moab::Interface::dimension_from_handle(), ErrorCode, moab::Interface::get_adjacencies(), MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, pstatus_tag(), moab::Range::rbegin(), moab::Range::size(), moab::Interface::tag_get_data(), moab::Interface::tag_set_data(), and moab::Interface::UNION.

Referenced by set_pstatus_entities().

◆ set_rank()

void moab::ParallelComm::set_rank ( unsigned int  r)
inline

set rank for this pcomm; USED FOR TESTING ONLY!

Definition at line 1662 of file ParallelComm.hpp.

1663 {
1664  procConfig.proc_rank( r );
1665  if( procConfig.proc_size() < r ) procConfig.proc_size( r + 1 );
1666 }

References moab::ProcConfig::proc_rank(), moab::ProcConfig::proc_size(), and procConfig.

◆ set_recv_request()

void moab::ParallelComm::set_recv_request ( int  n_request)
inline

Definition at line 1702 of file ParallelComm.hpp.

1703 {
1704  recvReqs.resize( n_request, MPI_REQUEST_NULL );
1705 }

References recvReqs.

◆ set_send_request()

void moab::ParallelComm::set_send_request ( int  n_request)
inline

Definition at line 1697 of file ParallelComm.hpp.

1698 {
1699  sendReqs.resize( n_request, MPI_REQUEST_NULL );
1700 }

References sendReqs.

◆ set_sharing_data()

ErrorCode moab::ParallelComm::set_sharing_data ( EntityHandle  ent,
unsigned char  pstatus,
int  old_nump,
int  new_nump,
int *  ps,
EntityHandle *  hs 
)
private

Definition at line 6654 of file ParallelComm.cpp.

6660 {
6661  // If new nump is less than 3, the entity is no longer mutishared
6662  if( old_nump > 2 && ( pstatus & PSTATUS_MULTISHARED ) && new_nump < 3 )
6663  {
6664  // Unset multishared flag
6665  pstatus ^= PSTATUS_MULTISHARED;
6666  }
6667 
6668  // Check for consistency in input data
6669  // DBG
6670  /* bool con1 = ((new_nump == 2 && pstatus&PSTATUS_SHARED && !(pstatus&PSTATUS_MULTISHARED)) ||
6671  (new_nump > 2 && pstatus&PSTATUS_SHARED && pstatus&PSTATUS_MULTISHARED)); bool con2 =
6672  (!(pstatus&PSTATUS_GHOST) || pstatus&PSTATUS_SHARED); bool con3 = (new_nump < 3 ||
6673  (pstatus&PSTATUS_NOT_OWNED && ps[0] != (int)rank()) || (!(pstatus&PSTATUS_NOT_OWNED) && ps[0]
6674  == (int)rank())); std::cout<<"current rank = "<<rank()<<std::endl; std::cout<<"condition
6675  1::"<<con1<<std::endl; std::cout<<"condition 2::"<<con2<<std::endl; std::cout<<"condition
6676  3::"<<con3<<std::endl;*/
6677 
6678  // DBG
6679 
6680  assert( new_nump > 1 &&
6681  ( ( new_nump == 2 && pstatus & PSTATUS_SHARED &&
6682  !( pstatus & PSTATUS_MULTISHARED ) ) || // If <= 2 must not be multishared
6683  ( new_nump > 2 && pstatus & PSTATUS_SHARED &&
6684  pstatus & PSTATUS_MULTISHARED ) ) && // If > 2 procs, must be multishared
6685  ( !( pstatus & PSTATUS_GHOST ) || pstatus & PSTATUS_SHARED ) && // If ghost, it must also be shared
6686  ( new_nump < 3 ||
6687  ( pstatus & PSTATUS_NOT_OWNED && ps[0] != (int)rank() ) || // I'm not owner and first proc not me
6688  ( !( pstatus & PSTATUS_NOT_OWNED ) && ps[0] == (int)rank() ) ) // I'm owner and first proc is me
6689  );
6690 
6691 #ifndef NDEBUG
6692  {
6693  // Check for duplicates in proc list
6694  std::set< unsigned int > dumprocs;
6695  int dp = 0;
6696  for( ; dp < old_nump && -1 != ps[dp]; dp++ )
6697  dumprocs.insert( ps[dp] );
6698  assert( dp == (int)dumprocs.size() );
6699  }
6700 #endif
6701 
6702  ErrorCode result;
6703  // Reset any old data that needs to be
6704  if( old_nump > 2 && new_nump < 3 )
6705  {
6706  // Need to remove multishared tags
6707  result = mbImpl->tag_delete_data( sharedps_tag(), &ent, 1 );
6708  MB_CHK_SET_ERR( result, "set_sharing_data:1" );
6709  result = mbImpl->tag_delete_data( sharedhs_tag(), &ent, 1 );
6710  MB_CHK_SET_ERR( result, "set_sharing_data:2" );
6711  // if (new_nump < 2)
6712  // pstatus = 0x0;
6713  // else if (ps[0] != (int)proc_config().proc_rank())
6714  // pstatus |= PSTATUS_NOT_OWNED;
6715  }
6716  else if( ( old_nump < 3 && new_nump > 2 ) || ( old_nump > 1 && new_nump == 1 ) )
6717  {
6718  // Reset sharedp and sharedh tags
6719  int tmp_p = -1;
6720  EntityHandle tmp_h = 0;
6721  result = mbImpl->tag_set_data( sharedp_tag(), &ent, 1, &tmp_p );
6722  MB_CHK_SET_ERR( result, "set_sharing_data:3" );
6723  result = mbImpl->tag_set_data( sharedh_tag(), &ent, 1, &tmp_h );
6724  MB_CHK_SET_ERR( result, "set_sharing_data:4" );
6725  }
6726 
6727  assert( "check for multishared/owner I'm first proc" &&
6728  ( !( pstatus & PSTATUS_MULTISHARED ) || ( pstatus & ( PSTATUS_NOT_OWNED | PSTATUS_GHOST ) ) ||
6729  ( ps[0] == (int)rank() ) ) &&
6730  "interface entities should have > 1 proc" && ( !( pstatus & PSTATUS_INTERFACE ) || new_nump > 1 ) &&
6731  "ghost entities should have > 1 proc" && ( !( pstatus & PSTATUS_GHOST ) || new_nump > 1 ) );
6732 
6733  // Now set new data
6734  if( new_nump > 2 )
6735  {
6736  result = mbImpl->tag_set_data( sharedps_tag(), &ent, 1, ps );
6737  MB_CHK_SET_ERR( result, "set_sharing_data:5" );
6738  result = mbImpl->tag_set_data( sharedhs_tag(), &ent, 1, hs );
6739  MB_CHK_SET_ERR( result, "set_sharing_data:6" );
6740  }
6741  else
6742  {
6743  unsigned int j = ( ps[0] == (int)procConfig.proc_rank() ? 1 : 0 );
6744  assert( -1 != ps[j] );
6745  result = mbImpl->tag_set_data( sharedp_tag(), &ent, 1, ps + j );
6746  MB_CHK_SET_ERR( result, "set_sharing_data:7" );
6747  result = mbImpl->tag_set_data( sharedh_tag(), &ent, 1, hs + j );
6748  MB_CHK_SET_ERR( result, "set_sharing_data:8" );
6749  }
6750 
6751  result = mbImpl->tag_set_data( pstatus_tag(), &ent, 1, &pstatus );
6752  MB_CHK_SET_ERR( result, "set_sharing_data:9" );
6753 
6754  if( old_nump > 1 && new_nump < 2 ) sharedEnts.erase( ent );
6755 
6756  return result;
6757 }

References ErrorCode, MB_CHK_SET_ERR, mbImpl, moab::ProcConfig::proc_rank(), procConfig, PSTATUS_GHOST, PSTATUS_INTERFACE, PSTATUS_MULTISHARED, PSTATUS_NOT_OWNED, PSTATUS_SHARED, pstatus_tag(), rank(), sharedEnts, sharedh_tag(), sharedhs_tag(), sharedp_tag(), sharedps_tag(), moab::Interface::tag_delete_data(), and moab::Interface::tag_set_data().

Referenced by check_clean_iface(), and update_remote_data().

◆ set_size()

void moab::ParallelComm::set_size ( unsigned int  r)
inline

set rank for this pcomm; USED FOR TESTING ONLY!

Definition at line 1668 of file ParallelComm.hpp.

1669 {
1670  procConfig.proc_size( s );
1671 }

References moab::ProcConfig::proc_size(), and procConfig.

◆ settle_intersection_points()

ErrorCode moab::ParallelComm::settle_intersection_points ( Range &  edges,
Range &  shared_edges_owned,
std::vector< std::vector< EntityHandle > * > &  extraNodesVec,
double  tolerance 
)

Definition at line 9370 of file ParallelComm.cpp.

9374 {
9375  // The index of an edge in the edges Range will give the index for extraNodesVec
9376  // the strategy of this follows exchange tags strategy:
9377  ErrorCode result;
9378  int success;
9379 
9380  myDebug->tprintf( 1, "Entering settle_intersection_points\n" );
9381 
9382  // Get all procs interfacing to this proc
9383  std::set< unsigned int > exch_procs;
9384  result = get_comm_procs( exch_procs );
9385 
9386  // Post ghost irecv's for all interface procs
9387  // Index requests the same as buffer/sharing procs indices
9388  std::vector< MPI_Request > recv_intx_reqs( 3 * buffProcs.size(), MPI_REQUEST_NULL );
9389  std::vector< unsigned int >::iterator sit;
9390  int ind;
9391 
9392  reset_all_buffers();
9393  int incoming = 0;
9394 
9395  for( ind = 0, sit = buffProcs.begin(); sit != buffProcs.end(); ++sit, ind++ )
9396  {
9397  incoming++;
9398  PRINT_DEBUG_IRECV( *sit, procConfig.proc_rank(), remoteOwnedBuffs[ind]->mem_ptr, INITIAL_BUFF_SIZE,
9399  MB_MESG_TAGS_SIZE, incoming );
9400 
9401  success = MPI_Irecv( remoteOwnedBuffs[ind]->mem_ptr, INITIAL_BUFF_SIZE, MPI_UNSIGNED_CHAR, *sit,
9402  MB_MESG_TAGS_SIZE, procConfig.proc_comm(), &recv_intx_reqs[3 * ind] );
9403  if( success != MPI_SUCCESS )
9404  {
9405  MB_SET_ERR( MB_FAILURE, "Failed to post irecv in settle intersection point" );
9406  }
9407  }
9408 
9409  // Pack and send intersection points from this proc to others
9410  // Make sendReqs vector to simplify initialization
9411  sendReqs.resize( 3 * buffProcs.size(), MPI_REQUEST_NULL );
9412 
9413  // Take all shared entities if incoming list is empty
9414  Range& entities = shared_edges_owned;
9415 
9416  int dum_ack_buff;
9417 
9418  for( ind = 0, sit = buffProcs.begin(); sit != buffProcs.end(); ++sit, ind++ )
9419  {
9420  Range edges_to_send = entities;
9421 
9422  // Get ents shared by proc *sit
9423  result = filter_pstatus( edges_to_send, PSTATUS_SHARED, PSTATUS_AND, *sit );
9424  MB_CHK_SET_ERR( result, "Failed pstatus AND check" );
9425 
9426  // Remote nonowned entities; not needed, edges are already owned by this proc
9427 
9428  // Pack the data
9429  // Reserve space on front for size and for initial buff size
9430  Buffer* buff = localOwnedBuffs[ind];
9431  buff->reset_ptr( sizeof( int ) );
9432 
9433  /*result = pack_intx_points(edges_to_send, edges, extraNodesVec,
9434  localOwnedBuffs[ind], *sit);*/
9435 
9436  // Count first data, and see if it is enough room?
9437  // Send the remote handles
9438  std::vector< EntityHandle > dum_remote_edges( edges_to_send.size() );
9439  /*
9440  * get_remote_handles(const bool store_remote_handles,
9441  EntityHandle *from_vec,
9442  EntityHandle *to_vec_tmp,
9443  int num_ents, int to_proc,
9444  const std::vector<EntityHandle> &new_ents);
9445  */
9446  // We are sending count, num edges, remote edges handles, and then, for each edge:
9447  // -- nb intx points, 3*nbintPointsforEdge "doubles"
9448  std::vector< EntityHandle > dum_vec;
9449  result = get_remote_handles( true, edges_to_send, dum_remote_edges.data(), *sit, dum_vec );
9450  MB_CHK_SET_ERR( result, "Failed to get remote handles" );
9451  int count = 4; // Size of data
9452  count += sizeof( int ) * (int)edges_to_send.size();
9453  count += sizeof( EntityHandle ) * (int)edges_to_send.size(); // We will send the remote handles
9454  for( Range::iterator eit = edges_to_send.begin(); eit != edges_to_send.end(); ++eit )
9455  {
9456  EntityHandle edge = *eit;
9457  unsigned int indx = edges.find( edge ) - edges.begin();
9458  std::vector< EntityHandle >& intx_nodes = *( extraNodesVec[indx] );
9459  count += (int)intx_nodes.size() * 3 * sizeof( double ); // 3 integer for each entity handle
9460  }
9461  //
9462  buff->check_space( count );
9463  PACK_INT( buff->buff_ptr, edges_to_send.size() );
9464  PACK_EH( buff->buff_ptr, dum_remote_edges.data(), dum_remote_edges.size() );
9465  for( Range::iterator eit = edges_to_send.begin(); eit != edges_to_send.end(); ++eit )
9466  {
9467  EntityHandle edge = *eit;
9468  // Pack the remote edge
9469  unsigned int indx = edges.find( edge ) - edges.begin();
9470  std::vector< EntityHandle >& intx_nodes = *( extraNodesVec[indx] );
9471  PACK_INT( buff->buff_ptr, intx_nodes.size() );
9472 
9473  result = mbImpl->get_coords( intx_nodes.data(), intx_nodes.size(), (double*)buff->buff_ptr );
9474  MB_CHK_SET_ERR( result, "Failed to get coords" );
9475  buff->buff_ptr += 3 * sizeof( double ) * intx_nodes.size();
9476  }
9477 
9478  // Done packing the intx points and remote edges
9479  buff->set_stored_size();
9480 
9481  // Now send it
9482  result = send_buffer( *sit, localOwnedBuffs[ind], MB_MESG_TAGS_SIZE, sendReqs[3 * ind],
9483  recv_intx_reqs[3 * ind + 2], &dum_ack_buff, incoming );
9484  MB_CHK_SET_ERR( result, "Failed to send buffer" );
9485  }
9486 
9487  // Receive/unpack intx points
9488  while( incoming )
9489  {
9490  MPI_Status status;
9491  int index_in_recv_requests;
9492  PRINT_DEBUG_WAITANY( recv_intx_reqs, MB_MESG_TAGS_SIZE, procConfig.proc_rank() );
9493  success = MPI_Waitany( 3 * buffProcs.size(), recv_intx_reqs.data(), &index_in_recv_requests, &status );
9494  if( MPI_SUCCESS != success )
9495  {
9496  MB_SET_ERR( MB_FAILURE, "Failed in waitany in ghost exchange" );
9497  }
9498  // Processor index in the list is divided by 3
9499  ind = index_in_recv_requests / 3;
9500 
9501  PRINT_DEBUG_RECD( status );
9502 
9503  // OK, received something; decrement incoming counter
9504  incoming--;
9505 
9506  bool done = false;
9507  result = recv_buffer( MB_MESG_TAGS_SIZE, status, remoteOwnedBuffs[ind],
9508  recv_intx_reqs[3 * ind + 1], // This is for receiving the second message
9509  recv_intx_reqs[3 * ind + 2], // This would be for ack, but it is not
9510  // used; consider removing it
9511  incoming, localOwnedBuffs[ind],
9512  sendReqs[3 * ind + 1], // Send request for sending the second message
9513  sendReqs[3 * ind + 2], // This is for sending the ack
9514  done );
9515  MB_CHK_SET_ERR( result, "Failed to resize recv buffer" );
9516  if( done )
9517  {
9518  Buffer* buff = remoteOwnedBuffs[ind];
9519  buff->reset_ptr( sizeof( int ) );
9520  /*result = unpack_tags(remoteOwnedBuffs[ind/2]->buff_ptr, dum_vec, true,
9521  buffProcs[ind/2]);*/
9522  // Unpack now the edges and vertex info; compare with the existing vertex positions
9523 
9524  int num_edges;
9525 
9526  UNPACK_INT( buff->buff_ptr, num_edges );
9527  std::vector< EntityHandle > rec_edges;
9528  rec_edges.resize( num_edges );
9529  UNPACK_EH( buff->buff_ptr, rec_edges.data(), num_edges );
9530  for( int i = 0; i < num_edges; i++ )
9531  {
9532  EntityHandle edge = rec_edges[i];
9533  unsigned int indx = edges.find( edge ) - edges.begin();
9534  std::vector< EntityHandle >& intx_nodes = *( extraNodesVec[indx] );
9535  // Now get the number of nodes on this (now local) edge
9536  int nverts;
9537  UNPACK_INT( buff->buff_ptr, nverts );
9538  std::vector< double > pos_from_owner;
9539  pos_from_owner.resize( 3 * nverts );
9540  UNPACK_DBLS( buff->buff_ptr, pos_from_owner.data(), 3 * nverts );
9541  std::vector< double > current_positions( 3 * intx_nodes.size() );
9542  result = mbImpl->get_coords( intx_nodes.data(), intx_nodes.size(), current_positions.data() );
9543  MB_CHK_SET_ERR( result, "Failed to get current positions" );
9544  // Now, look at what we have in current pos, compare to pos from owner, and reset
9545  for( int k = 0; k < (int)intx_nodes.size(); k++ )
9546  {
9547  double* pk = &current_positions[3 * k];
9548  // Take the current pos k, and settle among the ones from owner:
9549  bool found = false;
9550  for( int j = 0; j < nverts && !found; j++ )
9551  {
9552  double* pj = &pos_from_owner[3 * j];
9553  double dist2 = ( pk[0] - pj[0] ) * ( pk[0] - pj[0] ) + ( pk[1] - pj[1] ) * ( pk[1] - pj[1] ) +
9554  ( pk[2] - pj[2] ) * ( pk[2] - pj[2] );
9555  if( dist2 < tolerance )
9556  {
9557  pk[0] = pj[0];
9558  pk[1] = pj[1];
9559  pk[2] = pj[2]; // Correct it!
9560  found = true;
9561  break;
9562  }
9563  }
9564  if( !found )
9565  {
9566 #ifndef NDEBUG
9567  std::cout << " pk:" << pk[0] << " " << pk[1] << " " << pk[2] << " not found \n";
9568 #endif
9569  result = MB_FAILURE;
9570  }
9571  }
9572  // After we are done resetting, we can set the new positions of nodes:
9573  result = mbImpl->set_coords( intx_nodes.data(), (int)intx_nodes.size(), current_positions.data() );
9574  MB_CHK_SET_ERR( result, "Failed to set new current positions" );
9575  }
9576  }
9577  }
9578 
9579  // OK, now wait
9580  if( myDebug->get_verbosity() == 5 )
9581  {
9582  success = MPI_Barrier( procConfig.proc_comm() );
9583  }
9584  else
9585  {
9586  MPI_Status status[3 * MAX_SHARING_PROCS];
9587  success = MPI_Waitall( 3 * buffProcs.size(), sendReqs.data(), status );
9588  }
9589  if( MPI_SUCCESS != success )
9590  {
9591  MB_SET_ERR( MB_FAILURE, "Failure in waitall in tag exchange" );
9592  }
9593 
9594  myDebug->tprintf( 1, "Exiting settle_intersection_points" );
9595 
9596  return MB_SUCCESS;
9597 }

References moab::Range::begin(), moab::ParallelComm::Buffer::buff_ptr, buffProcs, moab::ParallelComm::Buffer::check_space(), moab::Range::end(), ErrorCode, filter_pstatus(), moab::Range::find(), get_comm_procs(), moab::Interface::get_coords(), get_remote_handles(), moab::DebugOutput::get_verbosity(), INITIAL_BUFF_SIZE, localOwnedBuffs, MAX_SHARING_PROCS, MB_CHK_SET_ERR, moab::MB_MESG_TAGS_SIZE, MB_SET_ERR, MB_SUCCESS, mbImpl, myDebug, moab::PACK_EH(), moab::PACK_INT(), PRINT_DEBUG_IRECV, PRINT_DEBUG_RECD, PRINT_DEBUG_WAITANY, moab::ProcConfig::proc_comm(), moab::ProcConfig::proc_rank(), procConfig, PSTATUS_AND, PSTATUS_SHARED, recv_buffer(), remoteOwnedBuffs, reset_all_buffers(), moab::ParallelComm::Buffer::reset_ptr(), send_buffer(), sendReqs, moab::Interface::set_coords(), moab::ParallelComm::Buffer::set_stored_size(), moab::Range::size(), moab::tolerance, moab::DebugOutput::tprintf(), moab::UNPACK_DBLS(), moab::UNPACK_EH(), and moab::UNPACK_INT().

◆ sharedh_tag()

Tag moab::ParallelComm::sharedh_tag ( )

return sharedh tag

Definition at line 8253 of file ParallelComm.cpp.

8254 {
8255  if( !sharedhTag )
8256  {
8257  EntityHandle def_val = 0;
8258  ErrorCode result = mbImpl->tag_get_handle( PARALLEL_SHARED_HANDLE_TAG_NAME, 1, MB_TYPE_HANDLE, sharedhTag,
8259  MB_TAG_DENSE | MB_TAG_CREAT, &def_val );
8260  if( MB_SUCCESS != result ) return 0;
8261  }
8262 
8263  return sharedhTag;
8264 }

References ErrorCode, MB_SUCCESS, MB_TAG_CREAT, MB_TAG_DENSE, MB_TYPE_HANDLE, mbImpl, PARALLEL_SHARED_HANDLE_TAG_NAME, sharedhTag, and moab::Interface::tag_get_handle().

Referenced by check_clean_iface(), check_sent_ents(), clean_shared_tags(), correct_thin_ghost_layers(), get_owner_handle(), get_owning_part(), get_shared_proc_tags(), get_sharing_data(), get_sharing_parts(), set_sharing_data(), and update_remote_data_old().

◆ sharedhs_tag()

Tag moab::ParallelComm::sharedhs_tag ( )

return sharedhs tag

Definition at line 8267 of file ParallelComm.cpp.

8268 {
8269  if( !sharedhsTag )
8270  {
8271  ErrorCode result = mbImpl->tag_get_handle( PARALLEL_SHARED_HANDLES_TAG_NAME, MAX_SHARING_PROCS, MB_TYPE_HANDLE,
8272  sharedhsTag, MB_TAG_SPARSE | MB_TAG_CREAT );
8273  if( MB_SUCCESS != result ) return 0;
8274  }
8275 
8276  return sharedhsTag;
8277 }

References ErrorCode, MAX_SHARING_PROCS, MB_SUCCESS, MB_TAG_CREAT, MB_TAG_SPARSE, MB_TYPE_HANDLE, mbImpl, PARALLEL_SHARED_HANDLES_TAG_NAME, sharedhsTag, and moab::Interface::tag_get_handle().

Referenced by check_clean_iface(), check_sent_ents(), correct_thin_ghost_layers(), get_owner_handle(), get_owning_part(), get_shared_proc_tags(), get_sharing_data(), get_sharing_parts(), set_sharing_data(), and update_remote_data_old().

◆ sharedp_tag()

Tag moab::ParallelComm::sharedp_tag ( )

return sharedp tag

Definition at line 8226 of file ParallelComm.cpp.

8227 {
8228  if( !sharedpTag )
8229  {
8230  int def_val = -1;
8231  ErrorCode result = mbImpl->tag_get_handle( PARALLEL_SHARED_PROC_TAG_NAME, 1, MB_TYPE_INTEGER, sharedpTag,
8232  MB_TAG_DENSE | MB_TAG_CREAT, &def_val );
8233  if( MB_SUCCESS != result ) return 0;
8234  }
8235 
8236  return sharedpTag;
8237 }

References ErrorCode, MB_SUCCESS, MB_TAG_CREAT, MB_TAG_DENSE, MB_TYPE_INTEGER, mbImpl, PARALLEL_SHARED_PROC_TAG_NAME, sharedpTag, and moab::Interface::tag_get_handle().

Referenced by check_clean_iface(), check_sent_ents(), clean_shared_tags(), correct_thin_ghost_layers(), filter_pstatus(), get_interface_procs(), get_owner_handle(), get_owning_part(), get_shared_proc_tags(), get_sharing_data(), get_sharing_parts(), is_iface_proc(), pack_entities(), set_sharing_data(), and update_remote_data_old().

◆ sharedps_tag()

Tag moab::ParallelComm::sharedps_tag ( )

return sharedps tag

Definition at line 8240 of file ParallelComm.cpp.

8241 {
8242  if( !sharedpsTag )
8243  {
8244  ErrorCode result = mbImpl->tag_get_handle( PARALLEL_SHARED_PROCS_TAG_NAME, MAX_SHARING_PROCS, MB_TYPE_INTEGER,
8245  sharedpsTag, MB_TAG_SPARSE | MB_TAG_CREAT );
8246  if( MB_SUCCESS != result ) return 0;
8247  }
8248 
8249  return sharedpsTag;
8250 }

References ErrorCode, MAX_SHARING_PROCS, MB_SUCCESS, MB_TAG_CREAT, MB_TAG_SPARSE, MB_TYPE_INTEGER, mbImpl, PARALLEL_SHARED_PROCS_TAG_NAME, sharedpsTag, and moab::Interface::tag_get_handle().

Referenced by check_clean_iface(), check_sent_ents(), correct_thin_ghost_layers(), filter_pstatus(), get_interface_procs(), get_owner_handle(), get_owning_part(), get_shared_proc_tags(), get_sharing_data(), get_sharing_parts(), is_iface_proc(), set_sharing_data(), and update_remote_data_old().

◆ size()

unsigned moab::ParallelComm::size ( ) const
inline

Definition at line 648 of file ParallelComm.hpp.

649  {
650  return proc_config().proc_size();
651  }

References proc_config(), and moab::ProcConfig::proc_size().

Referenced by assign_global_ids(), augment_default_sets_with_ghosts(), moab::ScdInterface::construct_box(), exchange_ghost_cells(), exchange_owned_mesh(), gather_data(), moab::NestedRefine::generate_hm(), get_sent_ents(), hcFilter(), iMOAB_LoadMesh(), iMOAB_SendMesh(), iMOAB_SetDoubleTagStorageWithGid(), iMOAB_WriteLocalMesh(), moab::DataCoupler::interpolate(), laplacianFilter(), moab::ReadParallel::load_file(), moab::DataCoupler::locate_points(), main(), moab::ParallelMergeMesh::PartitionGlobalBox(), perform_laplacian_smoothing(), perform_lloyd_relaxation(), moab::LloydSmoother::perform_smooth(), moab::ParallelMergeMesh::PerformMerge(), moab::ParallelMergeMesh::PopulateMySkinEnts(), DeformMeshRemap::read_file(), resolve_shared_ents(), RuntimeContext::RuntimeContext(), send_entities(), send_recv_entities(), and tag_shared_verts().

◆ tag_iface_entities()

ErrorCode moab::ParallelComm::tag_iface_entities ( )
private

Set pstatus tag interface bit on entities in sets passed in.

Definition at line 4498 of file ParallelComm.cpp.

4499 {
4500  ErrorCode result = MB_SUCCESS;
4501  Range iface_ents, tmp_ents, rmv_ents;
4502  std::vector< unsigned char > pstat;
4503  unsigned char set_pstat;
4504  Range::iterator rit2;
4505  unsigned int i;
4506 
4507  for( Range::iterator rit = interfaceSets.begin(); rit != interfaceSets.end(); ++rit )
4508  {
4509  iface_ents.clear();
4510 
4511  result = mbImpl->get_entities_by_handle( *rit, iface_ents );
4512  MB_CHK_SET_ERR( result, "Failed to get interface set contents" );
4513  pstat.resize( iface_ents.size() );
4514  result = mbImpl->tag_get_data( pstatus_tag(), iface_ents, pstat.data() );
4515  MB_CHK_SET_ERR( result, "Failed to get pstatus values for interface set entities" );
4516  result = mbImpl->tag_get_data( pstatus_tag(), &( *rit ), 1, &set_pstat );
4517  MB_CHK_SET_ERR( result, "Failed to get pstatus values for interface set" );
4518  rmv_ents.clear();
4519  for( rit2 = iface_ents.begin(), i = 0; rit2 != iface_ents.end(); ++rit2, i++ )
4520  {
4521  if( !( pstat[i] & PSTATUS_INTERFACE ) )
4522  {
4523  rmv_ents.insert( *rit2 );
4524  pstat[i] = 0x0;
4525  }
4526  }
4527  result = mbImpl->remove_entities( *rit, rmv_ents );
4528  MB_CHK_SET_ERR( result, "Failed to remove entities from interface set" );
4529 
4530  if( !( set_pstat & PSTATUS_NOT_OWNED ) ) continue;
4531  // If we're here, we need to set the notowned status on (remaining) set contents
4532 
4533  // Remove rmv_ents from the contents list
4534  iface_ents = subtract( iface_ents, rmv_ents );
4535  // Compress the pstat vector (removing 0x0's)
4536  pstat.erase(
4537  std::remove_if(
4538  pstat.begin(),
4539  pstat.end(),
4540  [](unsigned char c) { return c == 0x0; }
4541  ),
4542  pstat.end()
4543  );
4544 
4545  // Fold the not_owned bit into remaining values
4546  unsigned int sz = iface_ents.size();
4547  for( i = 0; i < sz; i++ )
4548  pstat[i] |= PSTATUS_NOT_OWNED;
4549 
4550  // Set the tag on the entities
4551  result = mbImpl->tag_set_data( pstatus_tag(), iface_ents, pstat.data() );
4552  MB_CHK_SET_ERR( result, "Failed to set pstatus values for interface set entities" );
4553  }
4554 
4555  return MB_SUCCESS;
4556 }

References moab::Range::begin(), moab::Range::clear(), moab::Range::end(), ErrorCode, moab::Interface::get_entities_by_handle(), moab::Range::insert(), interfaceSets, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, PSTATUS_INTERFACE, PSTATUS_NOT_OWNED, pstatus_tag(), moab::Interface::remove_entities(), moab::Range::size(), moab::subtract(), moab::Interface::tag_get_data(), and moab::Interface::tag_set_data().

Referenced by exchange_ghost_cells().

◆ tag_shared_verts() [1/2]

ErrorCode moab::ParallelComm::tag_shared_verts ( TupleList &  shared_ents,
std::map< std::vector< int >, std::vector< EntityHandle > > &  proc_nvecs,
Range &  proc_verts,
unsigned int  i_extra = 1 
)

Definition at line 5398 of file ParallelComm.cpp.

5402 {
5403  Tag shp_tag, shps_tag, shh_tag, shhs_tag, pstat_tag;
5404  ErrorCode result = get_shared_proc_tags( shp_tag, shps_tag, shh_tag, shhs_tag, pstat_tag );
5405  MB_CHK_SET_ERR( result, "Failed to get shared proc tags in tag_shared_verts" );
5406 
5407  unsigned int j = 0, i = 0;
5408  std::vector< int > sharing_procs, sharing_procs2, tag_procs;
5409  std::vector< EntityHandle > sharing_handles, sharing_handles2, tag_lhandles, tag_rhandles;
5410  std::vector< unsigned char > pstatus;
5411 
5412  // Were on tuple j/2
5413  if( i_extra ) i += i_extra;
5414  while( j < 2 * shared_ents.get_n() )
5415  {
5416  // Count & accumulate sharing procs
5417  EntityHandle this_ent = shared_ents.vul_rd[j], other_ent = 0;
5418  int other_proc = -1;
5419  while( j < 2 * shared_ents.get_n() && shared_ents.vul_rd[j] == this_ent )
5420  {
5421  j++;
5422  // Shouldn't have same proc
5423  assert( shared_ents.vi_rd[i] != (int)procConfig.proc_rank() );
5424  // Grab the remote data if its not a dublicate
5425  if( shared_ents.vul_rd[j] != other_ent || shared_ents.vi_rd[i] != other_proc )
5426  {
5427  assert( 0 != shared_ents.vul_rd[j] );
5428  sharing_procs.push_back( shared_ents.vi_rd[i] );
5429  sharing_handles.push_back( shared_ents.vul_rd[j] );
5430  }
5431  other_proc = shared_ents.vi_rd[i];
5432  other_ent = shared_ents.vul_rd[j];
5433  j++;
5434  i += 1 + i_extra;
5435  }
5436 
5437  if( sharing_procs.size() > 1 )
5438  {
5439  // Add current proc/handle to list
5440  sharing_procs.push_back( procConfig.proc_rank() );
5441  sharing_handles.push_back( this_ent );
5442 
5443  // Sort sharing_procs and sharing_handles such that
5444  // sharing_procs is in ascending order. Use temporary
5445  // lists and binary search to re-order sharing_handles.
5446  sharing_procs2 = sharing_procs;
5447  std::sort( sharing_procs2.begin(), sharing_procs2.end() );
5448  sharing_handles2.resize( sharing_handles.size() );
5449  for( size_t k = 0; k < sharing_handles.size(); k++ )
5450  {
5451  size_t idx = std::lower_bound( sharing_procs2.begin(), sharing_procs2.end(), sharing_procs[k] ) -
5452  sharing_procs2.begin();
5453  sharing_handles2[idx] = sharing_handles[k];
5454  }
5455  sharing_procs.swap( sharing_procs2 );
5456  sharing_handles.swap( sharing_handles2 );
5457  }
5458 
5459  assert( sharing_procs.size() != 2 );
5460  proc_nvecs[sharing_procs].push_back( this_ent );
5461 
5462  unsigned char share_flag = PSTATUS_SHARED, ms_flag = ( PSTATUS_SHARED | PSTATUS_MULTISHARED );
5463  if( sharing_procs.size() == 1 )
5464  {
5465  tag_procs.push_back( sharing_procs[0] );
5466  tag_lhandles.push_back( this_ent );
5467  tag_rhandles.push_back( sharing_handles[0] );
5468  pstatus.push_back( share_flag );
5469  }
5470  else
5471  {
5472  // Pad lists
5473  // assert(sharing_procs.size() <= MAX_SHARING_PROCS);
5474  if( sharing_procs.size() > MAX_SHARING_PROCS )
5475  {
5476  std::cerr << "MAX_SHARING_PROCS exceeded for vertex " << this_ent << " on process "
5477  << proc_config().proc_rank() << std::endl;
5478  std::cerr.flush();
5479  MPI_Abort( proc_config().proc_comm(), 66 );
5480  }
5481  sharing_procs.resize( MAX_SHARING_PROCS, -1 );
5482  sharing_handles.resize( MAX_SHARING_PROCS, 0 );
5483  result = mbImpl->tag_set_data( shps_tag, &this_ent, 1, sharing_procs.data() );
5484  MB_CHK_SET_ERR( result, "Failed to set sharedps tag on shared vertex" );
5485  result = mbImpl->tag_set_data( shhs_tag, &this_ent, 1, sharing_handles.data() );
5486  MB_CHK_SET_ERR( result, "Failed to set sharedhs tag on shared vertex" );
5487  result = mbImpl->tag_set_data( pstat_tag, &this_ent, 1, &ms_flag );
5488  MB_CHK_SET_ERR( result, "Failed to set pstatus tag on shared vertex" );
5489  sharedEnts.insert( this_ent );
5490  }
5491 
5492  // Reset sharing proc(s) tags
5493  sharing_procs.clear();
5494  sharing_handles.clear();
5495  }
5496 
5497  if( !tag_procs.empty() )
5498  {
5499  result = mbImpl->tag_set_data( shp_tag, tag_lhandles.data(), tag_procs.size(), tag_procs.data() );
5500  MB_CHK_SET_ERR( result, "Failed to set sharedp tag on shared vertex" );
5501  result = mbImpl->tag_set_data( shh_tag, tag_lhandles.data(), tag_procs.size(), tag_rhandles.data() );
5502  MB_CHK_SET_ERR( result, "Failed to set sharedh tag on shared vertex" );
5503  result = mbImpl->tag_set_data( pstat_tag, tag_lhandles.data(), tag_procs.size(), pstatus.data() );
5504  MB_CHK_SET_ERR( result, "Failed to set pstatus tag on shared vertex" );
5505  for( std::vector< EntityHandle >::iterator vvt = tag_lhandles.begin(); vvt != tag_lhandles.end(); vvt++ )
5506  sharedEnts.insert( *vvt );
5507  }
5508 
5509 #ifndef NDEBUG
5510  // Shouldn't be any repeated entities in any of the vectors in proc_nvecs
5511  for( std::map< std::vector< int >, std::vector< EntityHandle > >::iterator mit = proc_nvecs.begin();
5512  mit != proc_nvecs.end(); ++mit )
5513  {
5514  std::vector< EntityHandle > tmp_vec = ( mit->second );
5515  std::sort( tmp_vec.begin(), tmp_vec.end() );
5516  std::vector< EntityHandle >::iterator vit = std::unique( tmp_vec.begin(), tmp_vec.end() );
5517  assert( vit == tmp_vec.end() );
5518  }
5519 #endif
5520 
5521  return MB_SUCCESS;
5522 }

References ErrorCode, moab::TupleList::get_n(), get_shared_proc_tags(), MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, proc_config(), moab::ProcConfig::proc_rank(), procConfig, PSTATUS_MULTISHARED, PSTATUS_SHARED, sharedEnts, moab::Interface::tag_set_data(), moab::TupleList::vi_rd, and moab::TupleList::vul_rd.

Referenced by resolve_shared_ents(), moab::ScdInterface::tag_shared_vertices(), and moab::ParallelMergeMesh::TagSharedElements().

◆ tag_shared_verts() [2/2]

ErrorCode moab::ParallelComm::tag_shared_verts ( TupleList &  shared_verts,
Range *  skin_ents,
std::map< std::vector< int >, std::vector< EntityHandle > > &  proc_nvecs,
Range &  proc_verts 
)
private

Definition at line 5524 of file ParallelComm.cpp.

5528 {
5529  Tag shp_tag, shps_tag, shh_tag, shhs_tag, pstat_tag;
5530  ErrorCode result = get_shared_proc_tags( shp_tag, shps_tag, shh_tag, shhs_tag, pstat_tag );
5531  MB_CHK_SET_ERR( result, "Failed to get shared proc tags in tag_shared_verts" );
5532 
5533  unsigned int j = 0, i = 0;
5534  std::vector< int > sharing_procs, sharing_procs2;
5535  std::vector< EntityHandle > sharing_handles, sharing_handles2, skin_verts( skin_ents[0].size() );
5536  for( Range::iterator rit = skin_ents[0].begin(); rit != skin_ents[0].end(); ++rit, i++ )
5537  skin_verts[i] = *rit;
5538  i = 0;
5539 
5540  while( j < 2 * shared_ents.get_n() )
5541  {
5542  // Count & accumulate sharing procs
5543  int this_idx = shared_ents.vi_rd[j];
5544  EntityHandle this_ent = skin_verts[this_idx];
5545  while( j < 2 * shared_ents.get_n() && shared_ents.vi_rd[j] == this_idx )
5546  {
5547  j++;
5548  // Shouldn't have same proc
5549  assert( shared_ents.vi_rd[j] != (int)procConfig.proc_rank() );
5550  sharing_procs.push_back( shared_ents.vi_rd[j++] );
5551  sharing_handles.push_back( shared_ents.vul_rd[i++] );
5552  }
5553 
5554  if( sharing_procs.size() > 1 )
5555  {
5556  // Add current proc/handle to list
5557  sharing_procs.push_back( procConfig.proc_rank() );
5558  sharing_handles.push_back( this_ent );
5559  }
5560 
5561  // Sort sharing_procs and sharing_handles such that
5562  // sharing_procs is in ascending order. Use temporary
5563  // lists and binary search to re-order sharing_handles.
5564  sharing_procs2 = sharing_procs;
5565  std::sort( sharing_procs2.begin(), sharing_procs2.end() );
5566  sharing_handles2.resize( sharing_handles.size() );
5567  for( size_t k = 0; k < sharing_handles.size(); k++ )
5568  {
5569  size_t idx = std::lower_bound( sharing_procs2.begin(), sharing_procs2.end(), sharing_procs[k] ) -
5570  sharing_procs2.begin();
5571  sharing_handles2[idx] = sharing_handles[k];
5572  }
5573  sharing_procs.swap( sharing_procs2 );
5574  sharing_handles.swap( sharing_handles2 );
5575 
5576  assert( sharing_procs.size() != 2 );
5577  proc_nvecs[sharing_procs].push_back( this_ent );
5578 
5579  unsigned char share_flag = PSTATUS_SHARED, ms_flag = ( PSTATUS_SHARED | PSTATUS_MULTISHARED );
5580  if( sharing_procs.size() == 1 )
5581  {
5582  result = mbImpl->tag_set_data( shp_tag, &this_ent, 1, sharing_procs.data() );
5583  MB_CHK_SET_ERR( result, "Failed to set sharedp tag on shared vertex" );
5584  result = mbImpl->tag_set_data( shh_tag, &this_ent, 1, sharing_handles.data() );
5585  MB_CHK_SET_ERR( result, "Failed to set sharedh tag on shared vertex" );
5586  result = mbImpl->tag_set_data( pstat_tag, &this_ent, 1, &share_flag );
5587  MB_CHK_SET_ERR( result, "Failed to set pstatus tag on shared vertex" );
5588  sharedEnts.insert( this_ent );
5589  }
5590  else
5591  {
5592  // Pad lists
5593  // assert(sharing_procs.size() <= MAX_SHARING_PROCS);
5594  if( sharing_procs.size() > MAX_SHARING_PROCS )
5595  {
5596  std::cerr << "MAX_SHARING_PROCS exceeded for vertex " << this_ent << " on process "
5597  << proc_config().proc_rank() << std::endl;
5598  std::cerr.flush();
5599  MPI_Abort( proc_config().proc_comm(), 66 );
5600  }
5601  sharing_procs.resize( MAX_SHARING_PROCS, -1 );
5602  sharing_handles.resize( MAX_SHARING_PROCS, 0 );
5603  result = mbImpl->tag_set_data( shps_tag, &this_ent, 1, sharing_procs.data() );
5604  MB_CHK_SET_ERR( result, "Failed to set sharedps tag on shared vertex" );
5605  result = mbImpl->tag_set_data( shhs_tag, &this_ent, 1, sharing_handles.data() );
5606  MB_CHK_SET_ERR( result, "Failed to set sharedhs tag on shared vertex" );
5607  result = mbImpl->tag_set_data( pstat_tag, &this_ent, 1, &ms_flag );
5608  MB_CHK_SET_ERR( result, "Failed to set pstatus tag on shared vertex" );
5609  sharedEnts.insert( this_ent );
5610  }
5611 
5612  // Reset sharing proc(s) tags
5613  sharing_procs.clear();
5614  sharing_handles.clear();
5615  }
5616 
5617 #ifndef NDEBUG
5618  // Shouldn't be any repeated entities in any of the vectors in proc_nvecs
5619  for( std::map< std::vector< int >, std::vector< EntityHandle > >::iterator mit = proc_nvecs.begin();
5620  mit != proc_nvecs.end(); ++mit )
5621  {
5622  std::vector< EntityHandle > tmp_vec = ( mit->second );
5623  std::sort( tmp_vec.begin(), tmp_vec.end() );
5624  std::vector< EntityHandle >::iterator vit = std::unique( tmp_vec.begin(), tmp_vec.end() );
5625  assert( vit == tmp_vec.end() );
5626  }
5627 #endif
5628 
5629  return MB_SUCCESS;
5630 }

References moab::Range::end(), ErrorCode, moab::TupleList::get_n(), get_shared_proc_tags(), MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, proc_config(), moab::ProcConfig::proc_rank(), procConfig, PSTATUS_MULTISHARED, PSTATUS_SHARED, sharedEnts, size(), moab::Interface::tag_set_data(), moab::TupleList::vi_rd, and moab::TupleList::vul_rd.

◆ unpack_adjacencies()

ErrorCode moab::ParallelComm::unpack_adjacencies ( unsigned char *&  buff_ptr,
Range &  entities,
const bool  store_handles,
const int  from_proc 
)
private

Definition at line 3569 of file ParallelComm.cpp.

3573 {
3574  return MB_FAILURE;
3575 }

◆ unpack_buffer()

ErrorCode moab::ParallelComm::unpack_buffer ( unsigned char *  buff_ptr,
const bool  store_remote_handles,
const int  from_proc,
const int  ind,
std::vector< std::vector< EntityHandle > > &  L1hloc,
std::vector< std::vector< EntityHandle > > &  L1hrem,
std::vector< std::vector< int > > &  L1p,
std::vector< EntityHandle > &  L2hloc,
std::vector< EntityHandle > &  L2hrem,
std::vector< unsigned int > &  L2p,
std::vector< EntityHandle > &  new_ents,
const bool  created_iface = false 
)

Definition at line 1497 of file ParallelComm.cpp.

1509 {
1510  unsigned char* tmp_buff = buff_ptr;
1511  ErrorCode result;
1512  result = unpack_entities( buff_ptr, store_remote_handles, ind, false, L1hloc, L1hrem, L1p, L2hloc, L2hrem, L2p,
1513  new_ents, created_iface );
1514  MB_CHK_SET_ERR( result, "Unpacking entities failed" );
1515  if( myDebug->get_verbosity() == 3 )
1516  {
1517  myDebug->tprintf( 4, "unpack_entities buffer space: %ld bytes.\n", (long int)( buff_ptr - tmp_buff ) );
1518  tmp_buff = buff_ptr;
1519  }
1520  result = unpack_sets( buff_ptr, new_ents, store_remote_handles, from_proc );
1521  MB_CHK_SET_ERR( result, "Unpacking sets failed" );
1522  if( myDebug->get_verbosity() == 3 )
1523  {
1524  myDebug->tprintf( 4, "unpack_sets buffer space: %ld bytes.\n", (long int)( buff_ptr - tmp_buff ) );
1525  tmp_buff = buff_ptr;
1526  }
1527  result = unpack_tags( buff_ptr, new_ents, store_remote_handles, from_proc );
1528  MB_CHK_SET_ERR( result, "Unpacking tags failed" );
1529  if( myDebug->get_verbosity() == 3 )
1530  {
1531  myDebug->tprintf( 4, "unpack_tags buffer space: %ld bytes.\n", (long int)( buff_ptr - tmp_buff ) );
1532  // tmp_buff = buff_ptr;
1533  }
1534 
1535  if( myDebug->get_verbosity() == 3 ) myDebug->print( 4, "\n" );
1536 
1537  return MB_SUCCESS;
1538 }

References ErrorCode, moab::DebugOutput::get_verbosity(), MB_CHK_SET_ERR, MB_SUCCESS, myDebug, moab::DebugOutput::print(), moab::DebugOutput::tprintf(), unpack_entities(), unpack_sets(), and unpack_tags().

Referenced by broadcast_entities(), exchange_owned_mesh(), moab::ParCommGraph::receive_mesh(), recv_entities(), recv_messages(), and scatter_entities().

◆ unpack_entities()

ErrorCode moab::ParallelComm::unpack_entities ( unsigned char *&  buff_ptr,
const bool  store_remote_handles,
const int  from_ind,
const bool  is_iface,
std::vector< std::vector< EntityHandle > > &  L1hloc,
std::vector< std::vector< EntityHandle > > &  L1hrem,
std::vector< std::vector< int > > &  L1p,
std::vector< EntityHandle > &  L2hloc,
std::vector< EntityHandle > &  L2hrem,
std::vector< unsigned int > &  L2p,
std::vector< EntityHandle > &  new_ents,
const bool  created_iface = false 
)

unpack entities in buff_ptr

Definition at line 2129 of file ParallelComm.cpp.

2141 {
2142  // General algorithm:
2143  // - unpack # entities
2144  // - save start of remote handle info, then scan forward to entity definition data
2145  // - for all vertices or entities w/ same # verts:
2146  // . get entity type, num ents, and (if !vert) # verts
2147  // . for each ent:
2148  // o get # procs/handles in remote handle info
2149  // o if # procs/handles > 2, check for already-created entity:
2150  // x get index of owner proc (1st in proc list), resize L1 list if nec
2151  // x look for already-arrived entity in L2 by owner handle
2152  // o if no existing entity:
2153  // x if iface, look for existing entity with same connect & type
2154  // x if none found, create vertex or element
2155  // x if !iface & multi-shared, save on L2
2156  // x if !iface, put new entity on new_ents list
2157  // o update proc/handle, pstatus tags, adjusting to put owner first if iface
2158  // o if !iface, save new handle on L1 for all sharing procs
2159 
2160  // Lists of handles/procs to return to sending/other procs
2161  // L1hloc[p], L1hrem[p]: handle pairs [h, h'], where h is the local proc handle
2162  // and h' is either the remote proc handle (if that is known) or
2163  // the owner proc handle (otherwise);
2164  // L1p[p]: indicates whether h is remote handle (= -1) or owner (rank of owner)
2165  // L2hloc, L2hrem: local/remote handles for entities shared by > 2 procs;
2166  // remote handles are on owning proc
2167  // L2p: owning procs for handles in L2hrem
2168 
2169  ErrorCode result;
2170  bool done = false;
2171  ReadUtilIface* ru = NULL;
2172 
2173  result = mbImpl->query_interface( ru );
2174  MB_CHK_SET_ERR( result, "Failed to get ReadUtilIface" );
2175 
2176  // 1. # entities = E
2177  int num_ents = 0;
2178  unsigned char* buff_save = buff_ptr;
2179  int i, j;
2180 
2181  if( store_remote_handles )
2182  {
2183  UNPACK_INT( buff_ptr, num_ents );
2184 
2185  buff_save = buff_ptr;
2186 
2187  // Save place where remote handle info starts, then scan forward to ents
2188  for( i = 0; i < num_ents; i++ )
2189  {
2190  UNPACK_INT( buff_ptr, j );
2191  if( j < 0 )
2192  {
2193  std::cout << "Should be non-negative # proc/handles.";
2194  return MB_FAILURE;
2195  }
2196 
2197  buff_ptr += j * ( sizeof( int ) + sizeof( EntityHandle ) );
2198  }
2199  }
2200 
2201  std::vector< EntityHandle > msg_ents;
2202 
2203  while( !done )
2204  {
2205  EntityType this_type = MBMAXTYPE;
2206  UNPACK_TYPE( buff_ptr, this_type );
2207  assert( this_type != MBENTITYSET );
2208 
2209  // MBMAXTYPE signifies end of entities data
2210  if( MBMAXTYPE == this_type ) break;
2211 
2212  // Get the number of ents
2213  int num_ents2, verts_per_entity = 0;
2214  UNPACK_INT( buff_ptr, num_ents2 );
2215 
2216  // Unpack the nodes per entity
2217  if( MBVERTEX != this_type && num_ents2 )
2218  {
2219  UNPACK_INT( buff_ptr, verts_per_entity );
2220  }
2221 
2222  std::vector< int > ps( MAX_SHARING_PROCS, -1 );
2223  std::vector< EntityHandle > hs( MAX_SHARING_PROCS, 0 );
2224  for( int e = 0; e < num_ents2; e++ )
2225  {
2226  // Check for existing entity, otherwise make new one
2227  EntityHandle new_h = 0;
2228  EntityHandle connect[CN::MAX_NODES_PER_ELEMENT];
2229  double coords[3];
2230  int num_ps = -1;
2231 
2232  //=======================================
2233  // Unpack all the data at once, to make sure the buffer pointers
2234  // are tracked correctly
2235  //=======================================
2236  if( store_remote_handles )
2237  {
2238  // Pointers to other procs/handles
2239  UNPACK_INT( buff_save, num_ps );
2240  if( 0 >= num_ps )
2241  {
2242  std::cout << "Shouldn't ever be fewer than 1 procs here." << std::endl;
2243  return MB_FAILURE;
2244  }
2245 
2246  UNPACK_INTS( buff_save, ps.data(), num_ps );
2247  UNPACK_EH( buff_save, hs.data(), num_ps );
2248  }
2249 
2250  if( MBVERTEX == this_type )
2251  {
2252  UNPACK_DBLS( buff_ptr, coords, 3 );
2253  }
2254  else
2255  {
2256  assert( verts_per_entity <= CN::MAX_NODES_PER_ELEMENT );
2257  UNPACK_EH( buff_ptr, connect, verts_per_entity );
2258 
2259  // Update connectivity to local handles
2260  result = get_local_handles( connect, verts_per_entity, msg_ents );
2261  MB_CHK_SET_ERR( result, "Failed to get local handles" );
2262  }
2263 
2264  //=======================================
2265  // Now, process that data; begin by finding an identical
2266  // entity, if there is one
2267  //=======================================
2268  if( store_remote_handles )
2269  {
2270  result = find_existing_entity( is_iface, ps[0], hs[0], num_ps, connect, verts_per_entity, this_type,
2271  L2hloc, L2hrem, L2p, new_h );
2272  MB_CHK_SET_ERR( result, "Failed to get existing entity" );
2273  }
2274 
2275  //=======================================
2276  // If we didn't find one, we'll have to create one
2277  //=======================================
2278  bool created_here = false;
2279  if( !new_h && !is_iface )
2280  {
2281  if( MBVERTEX == this_type )
2282  {
2283  // Create a vertex
2284  result = mbImpl->create_vertex( coords, new_h );
2285  MB_CHK_SET_ERR( result, "Failed to make new vertex" );
2286  }
2287  else
2288  {
2289  // Create the element
2290  result = mbImpl->create_element( this_type, connect, verts_per_entity, new_h );
2291  MB_CHK_SET_ERR( result, "Failed to make new element" );
2292 
2293  // Update adjacencies
2294  result = ru->update_adjacencies( new_h, 1, verts_per_entity, connect );
2295  MB_CHK_SET_ERR( result, "Failed to update adjacencies" );
2296  }
2297 
2298  // Should have a new handle now
2299  assert( new_h );
2300 
2301  created_here = true;
2302  }
2303 
2304  //=======================================
2305  // Take care of sharing data
2306  //=======================================
2307 
2308  // Need to save entities found in order, for interpretation of
2309  // later parts of this message
2310  if( !is_iface )
2311  {
2312  assert( new_h );
2313  msg_ents.push_back( new_h );
2314  }
2315 
2316  if( created_here ) new_ents.push_back( new_h );
2317 
2318  if( new_h && store_remote_handles )
2319  {
2320  unsigned char new_pstat = 0x0;
2321  if( is_iface )
2322  {
2323  new_pstat = PSTATUS_INTERFACE;
2324  // Here, lowest rank proc should be first
2325  int idx = std::min_element( ps.data(), ps.data() + num_ps ) - ps.data();
2326  if( idx )
2327  {
2328  std::swap( ps[0], ps[idx] );
2329  std::swap( hs[0], hs[idx] );
2330  }
2331  // Set ownership based on lowest rank; can't be in update_remote_data, because
2332  // there we don't know whether it resulted from ghosting or not
2333  if( ( num_ps > 1 && ps[0] != (int)rank() ) ) new_pstat |= PSTATUS_NOT_OWNED;
2334  }
2335  else if( created_here )
2336  {
2337  if( created_iface )
2338  new_pstat = PSTATUS_NOT_OWNED;
2339  else
2340  new_pstat = PSTATUS_GHOST | PSTATUS_NOT_OWNED;
2341  }
2342 
2343  // Update sharing data and pstatus, adjusting order if iface
2344  result = update_remote_data( new_h, ps.data(), hs.data(), num_ps, new_pstat );
2345  MB_CHK_SET_ERR( result, "unpack_entities" );
2346 
2347  // If a new multi-shared entity, save owner for subsequent lookup in L2 lists
2348  if( store_remote_handles && !is_iface && num_ps > 2 )
2349  {
2350  L2hrem.push_back( hs[0] );
2351  L2hloc.push_back( new_h );
2352  L2p.push_back( ps[0] );
2353  }
2354 
2355  // Need to send this new handle to all sharing procs
2356  if( !is_iface )
2357  {
2358  for( j = 0; j < num_ps; j++ )
2359  {
2360  if( ps[j] == (int)procConfig.proc_rank() ) continue;
2361  int idx = get_buffers( ps[j] );
2362  if( idx == (int)L1hloc.size() )
2363  {
2364  L1hloc.resize( idx + 1 );
2365  L1hrem.resize( idx + 1 );
2366  L1p.resize( idx + 1 );
2367  }
2368 
2369  // Don't bother adding if it's already in the list
2370  std::vector< EntityHandle >::iterator vit =
2371  std::find( L1hloc[idx].begin(), L1hloc[idx].end(), new_h );
2372  if( vit != L1hloc[idx].end() )
2373  {
2374  // If it's in the list but remote handle isn't known but we know
2375  // it, replace in the list
2376  if( L1p[idx][vit - L1hloc[idx].begin()] != -1 && hs[j] )
2377  {
2378  L1hrem[idx][vit - L1hloc[idx].begin()] = hs[j];
2379  L1p[idx][vit - L1hloc[idx].begin()] = -1;
2380  }
2381  else
2382  continue;
2383  }
2384  else
2385  {
2386  if( !hs[j] )
2387  {
2388  assert( -1 != ps[0] && num_ps > 2 );
2389  L1p[idx].push_back( ps[0] );
2390  L1hrem[idx].push_back( hs[0] );
2391  }
2392  else
2393  {
2394  assert(
2395  "either this remote handle isn't in the remote list, or "
2396  "it's for another proc" &&
2397  ( std::find( L1hrem[idx].begin(), L1hrem[idx].end(), hs[j] ) == L1hrem[idx].end() ||
2398  L1p[idx][std::find( L1hrem[idx].begin(), L1hrem[idx].end(), hs[j] ) -
2399  L1hrem[idx].begin()] != -1 ) );
2400  L1p[idx].push_back( -1 );
2401  L1hrem[idx].push_back( hs[j] );
2402  }
2403  L1hloc[idx].push_back( new_h );
2404  }
2405  }
2406  }
2407 
2408  assert( "Shouldn't be here for non-shared entities" && -1 != num_ps );
2409  std::fill( ps.data(), &ps[num_ps], -1 );
2410  std::fill( hs.data(), &hs[num_ps], 0 );
2411  }
2412  }
2413 
2414  myDebug->tprintf( 4, "Unpacked %d ents of type %s", num_ents2, CN::EntityTypeName( this_type ) );
2415  }
2416 
2417  myDebug->tprintf( 4, "Done unpacking entities.\n" );
2418 
2419  // Need to sort here, to enable searching
2420  // std::sort( new_ents.begin(), new_ents.end() );
2421 
2422  return MB_SUCCESS;
2423 }

References moab::Interface::create_element(), moab::Interface::create_vertex(), moab::CN::EntityTypeName(), ErrorCode, find_existing_entity(), get_buffers(), get_local_handles(), moab::CN::MAX_NODES_PER_ELEMENT, MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SUCCESS, MBENTITYSET, mbImpl, MBMAXTYPE, MBVERTEX, myDebug, moab::ProcConfig::proc_rank(), procConfig, PSTATUS_GHOST, PSTATUS_INTERFACE, PSTATUS_NOT_OWNED, moab::Interface::query_interface(), rank(), moab::DebugOutput::tprintf(), moab::UNPACK_DBLS(), moab::UNPACK_EH(), moab::UNPACK_INT(), moab::UNPACK_INTS(), moab::UNPACK_TYPE(), moab::ReadUtilIface::update_adjacencies(), and update_remote_data().

Referenced by exchange_ghost_cells(), and unpack_buffer().

◆ unpack_iface_entities()

ErrorCode moab::ParallelComm::unpack_iface_entities ( unsigned char *&  buff_ptr,
const int  from_proc,
const int  ind,
std::vector< EntityHandle > &  recd_ents 
)
private

for all the entities in the received buffer; for each, save entities in this instance which match connectivity, or zero if none found

◆ unpack_remote_handles() [1/2]

ErrorCode moab::ParallelComm::unpack_remote_handles ( unsigned int  from_proc,
const unsigned char *  buff_ptr,
std::vector< EntityHandle > &  L2hloc,
std::vector< EntityHandle > &  L2hrem,
std::vector< unsigned int > &  L2p 
)
inlineprivate

Definition at line 1651 of file ParallelComm.hpp.

1656 {
1657  // cast away const-ness, we won't be passing back a modified ptr
1658  unsigned char* tmp_buff = const_cast< unsigned char* >( buff_ptr );
1659  return unpack_remote_handles( from_proc, tmp_buff, L2hloc, L2hrem, L2p );
1660 }

References unpack_remote_handles().

◆ unpack_remote_handles() [2/2]

ErrorCode moab::ParallelComm::unpack_remote_handles ( unsigned int  from_proc,
unsigned char *&  buff_ptr,
std::vector< EntityHandle > &  L2hloc,
std::vector< EntityHandle > &  L2hrem,
std::vector< unsigned int > &  L2p 
)

Definition at line 7676 of file ParallelComm.cpp.

7681 {
7682  // Incoming remote handles; use to set remote handles
7683  int num_eh;
7684  UNPACK_INT( buff_ptr, num_eh );
7685 
7686  unsigned char* buff_proc = buff_ptr;
7687  buff_ptr += num_eh * sizeof( int );
7688  unsigned char* buff_rem = buff_ptr + num_eh * sizeof( EntityHandle );
7689  ErrorCode result;
7690  EntityHandle hpair[2], new_h;
7691  int proc;
7692  for( int i = 0; i < num_eh; i++ )
7693  {
7694  UNPACK_INT( buff_proc, proc );
7695  // Handles packed (local, remote), though here local is either on this
7696  // proc or owner proc, depending on value of proc (-1 = here, otherwise owner);
7697  // this is decoded in find_existing_entity
7698  UNPACK_EH( buff_ptr, hpair, 1 );
7699  UNPACK_EH( buff_rem, hpair + 1, 1 );
7700 
7701  if( -1 != proc )
7702  {
7703  result = find_existing_entity( false, proc, hpair[0], 3, NULL, 0, mbImpl->type_from_handle( hpair[1] ),
7704  L2hloc, L2hrem, L2p, new_h );
7705  MB_CHK_SET_ERR( result, "Didn't get existing entity" );
7706  if( new_h )
7707  hpair[0] = new_h;
7708  else
7709  hpair[0] = 0;
7710  }
7711  if( !( hpair[0] && hpair[1] ) ) return MB_FAILURE;
7712  int this_proc = from_proc;
7713  result = update_remote_data( hpair[0], &this_proc, hpair + 1, 1, 0 );
7714  MB_CHK_SET_ERR( result, "Failed to set remote data range on sent entities in ghost exchange" );
7715  }
7716 
7717  return MB_SUCCESS;
7718 }

References ErrorCode, find_existing_entity(), MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, moab::Interface::type_from_handle(), moab::UNPACK_EH(), moab::UNPACK_INT(), and update_remote_data().

Referenced by exchange_ghost_cells(), exchange_owned_mesh(), recv_entities(), recv_remote_handle_messages(), and unpack_remote_handles().

◆ unpack_sets()

ErrorCode moab::ParallelComm::unpack_sets ( unsigned char *&  buff_ptr,
std::vector< EntityHandle > &  entities,
const bool  store_handles,
const int  to_proc 
)
private

Definition at line 3407 of file ParallelComm.cpp.

3411 {
3412  // Now the sets; assume any sets the application wants to pass are in the entities list
3413  ErrorCode result;
3414 
3415  bool no_sets = ( entities.empty() || ( mbImpl->type_from_handle( *entities.rbegin() ) == MBENTITYSET ) );
3416 
3417  Range new_sets;
3418  int num_sets;
3419  UNPACK_INT( buff_ptr, num_sets );
3420 
3421  if( !num_sets ) return MB_SUCCESS;
3422 
3423  int i;
3424  Range::const_iterator rit;
3425  std::vector< EntityHandle > members;
3426  int num_ents;
3427  std::vector< unsigned int > options_vec( num_sets );
3428  // Option value
3429  if( num_sets ) UNPACK_VOID( buff_ptr, options_vec.data(), num_sets * sizeof( unsigned int ) );
3430 
3431  // Unpack parallel geometry unique id
3432  int n_uid;
3433  UNPACK_INT( buff_ptr, n_uid );
3434  if( n_uid > 0 && n_uid != num_sets )
3435  {
3436  std::cerr << "The number of Parallel geometry unique ids should be same." << std::endl;
3437  }
3438 
3439  if( n_uid > 0 )
3440  { // If parallel geometry unique id is packed
3441  std::vector< int > uids( n_uid );
3442  UNPACK_INTS( buff_ptr, uids.data(), n_uid );
3443 
3444  Tag uid_tag;
3445  result =
3446  mbImpl->tag_get_handle( "PARALLEL_UNIQUE_ID", 1, MB_TYPE_INTEGER, uid_tag, MB_TAG_SPARSE | MB_TAG_CREAT );
3447  MB_CHK_SET_ERR( result, "Failed to create parallel geometry unique id tag" );
3448 
3449  // Find existing sets
3450  for( i = 0; i < n_uid; i++ )
3451  {
3452  EntityHandle set_handle;
3453  Range temp_sets;
3454  void* tag_vals[] = { &uids[i] };
3455  if( uids[i] > 0 )
3456  {
3457  result = mbImpl->get_entities_by_type_and_tag( 0, MBENTITYSET, &uid_tag, tag_vals, 1, temp_sets );
3458  }
3459  if( !temp_sets.empty() )
3460  { // Existing set
3461  set_handle = *temp_sets.begin();
3462  }
3463  else
3464  { // Create a new set
3465  result = mbImpl->create_meshset( options_vec[i], set_handle );
3466  MB_CHK_SET_ERR( result, "Failed to create set in unpack" );
3467  result = mbImpl->tag_set_data( uid_tag, &set_handle, 1, &uids[i] );
3468  MB_CHK_SET_ERR( result, "Failed to set parallel geometry unique ids" );
3469  }
3470  new_sets.insert( set_handle );
3471  }
3472  }
3473  else
3474  {
3475  // Create sets
3476  for( i = 0; i < num_sets; i++ )
3477  {
3478  EntityHandle set_handle;
3479  result = mbImpl->create_meshset( options_vec[i], set_handle );
3480  MB_CHK_SET_ERR( result, "Failed to create set in unpack" );
3481 
3482  // Make sure new sets handles are monotonically increasing
3483  assert( set_handle > *new_sets.rbegin() );
3484  new_sets.insert( set_handle );
3485  }
3486  }
3487 
3488  std::copy( new_sets.begin(), new_sets.end(), std::back_inserter( entities ) );
3489  // Only need to sort if we came in with no sets on the end
3490  if( !no_sets ) std::sort( entities.begin(), entities.end() );
3491 
3492  for( rit = new_sets.begin(), i = 0; rit != new_sets.end(); ++rit, i++ )
3493  {
3494  // Unpack entities as vector, with length
3495  UNPACK_INT( buff_ptr, num_ents );
3496  members.resize( num_ents );
3497  if( num_ents ) UNPACK_EH( buff_ptr, members.data(), num_ents );
3498  result = get_local_handles( members.data(), num_ents, entities );
3499  MB_CHK_SET_ERR( result, "Failed to get local handles for ordered set contents" );
3500  result = mbImpl->add_entities( *rit, members.data(), num_ents );
3501  MB_CHK_SET_ERR( result, "Failed to add ents to ordered set in unpack" );
3502  }
3503 
3504  std::vector< int > num_pch( 2 * new_sets.size() );
3505  std::vector< int >::iterator vit;
3506  int tot_pch = 0;
3507  for( vit = num_pch.begin(); vit != num_pch.end(); ++vit )
3508  {
3509  UNPACK_INT( buff_ptr, *vit );
3510  tot_pch += *vit;
3511  }
3512 
3513  members.resize( tot_pch );
3514  UNPACK_EH( buff_ptr, members.data(), tot_pch );
3515  result = get_local_handles( members.data(), tot_pch, entities );
3516  MB_CHK_SET_ERR( result, "Failed to get local handle for parent/child sets" );
3517 
3518  int num = 0;
3519  EntityHandle* mem_ptr = members.data();
3520  for( rit = new_sets.begin(); rit != new_sets.end(); ++rit )
3521  {
3522  // Unpack parents/children
3523  int num_par = num_pch[num++], num_child = num_pch[num++];
3524  if( num_par + num_child )
3525  {
3526  for( i = 0; i < num_par; i++ )
3527  {
3528  assert( 0 != mem_ptr[i] );
3529  result = mbImpl->add_parent_meshset( *rit, mem_ptr[i] );
3530  MB_CHK_SET_ERR( result, "Failed to add parent to set in unpack" );
3531  }
3532  mem_ptr += num_par;
3533  for( i = 0; i < num_child; i++ )
3534  {
3535  assert( 0 != mem_ptr[i] );
3536  result = mbImpl->add_child_meshset( *rit, mem_ptr[i] );
3537  MB_CHK_SET_ERR( result, "Failed to add child to set in unpack" );
3538  }
3539  mem_ptr += num_child;
3540  }
3541  }
3542 
3543  // Unpack source handles
3544  Range dum_range;
3545  if( store_remote_handles && !new_sets.empty() )
3546  {
3547  UNPACK_RANGE( buff_ptr, dum_range );
3548  result = update_remote_data( new_sets, dum_range, from_proc, 0 );
3549  MB_CHK_SET_ERR( result, "Failed to set sharing data for sets" );
3550  }
3551 
3552  myDebug->tprintf( 4, "Done unpacking sets." );
3553 
3554  return MB_SUCCESS;
3555 }

References moab::Interface::add_child_meshset(), moab::Interface::add_entities(), moab::Interface::add_parent_meshset(), moab::Range::begin(), moab::Interface::create_meshset(), moab::Range::empty(), moab::Range::end(), ErrorCode, moab::Interface::get_entities_by_type_and_tag(), get_local_handles(), moab::Range::insert(), MB_CHK_SET_ERR, MB_SUCCESS, MB_TAG_CREAT, MB_TAG_SPARSE, MB_TYPE_INTEGER, MBENTITYSET, mbImpl, myDebug, moab::Range::rbegin(), moab::Range::size(), moab::Interface::tag_get_handle(), moab::Interface::tag_set_data(), moab::DebugOutput::tprintf(), moab::Interface::type_from_handle(), moab::UNPACK_EH(), moab::UNPACK_INT(), moab::UNPACK_INTS(), moab::UNPACK_RANGE(), moab::UNPACK_VOID(), and update_remote_data().

Referenced by unpack_buffer().

◆ unpack_tags()

ErrorCode moab::ParallelComm::unpack_tags ( unsigned char *&  buff_ptr,
std::vector< EntityHandle > &  entities,
const bool  store_handles,
const int  to_proc,
const MPI_Op *const  mpi_op = NULL 
)
private

Definition at line 3792 of file ParallelComm.cpp.

3797 {
3798  // Tags
3799  // Get all the tags
3800  // For dense tags, compute size assuming all entities have that tag
3801  // For sparse tags, get number of entities w/ that tag to compute size
3802 
3803  ErrorCode result;
3804 
3805  int num_tags;
3806  UNPACK_INT( buff_ptr, num_tags );
3807  std::vector< const void* > var_len_vals;
3808  std::vector< unsigned char > dum_vals;
3809  std::vector< EntityHandle > dum_ehvals;
3810 
3811  for( int i = 0; i < num_tags; i++ )
3812  {
3813  // Tag handle
3814  Tag tag_handle;
3815 
3816  // Size, data type
3817  int tag_size, tag_data_type, tag_type;
3818  UNPACK_INT( buff_ptr, tag_size );
3819  UNPACK_INT( buff_ptr, tag_type );
3820  UNPACK_INT( buff_ptr, tag_data_type );
3821 
3822  // Default value
3823  int def_val_size;
3824  UNPACK_INT( buff_ptr, def_val_size );
3825  void* def_val_ptr = NULL;
3826  if( def_val_size )
3827  {
3828  def_val_ptr = buff_ptr;
3829  buff_ptr += def_val_size;
3830  UPC( tag_size, " void" );
3831  }
3832 
3833  // Name
3834  int name_len;
3835  UNPACK_INT( buff_ptr, name_len );
3836  std::string tag_name( reinterpret_cast< char* >( buff_ptr ), name_len );
3837  buff_ptr += name_len;
3838  UPC( 64, " chars" );
3839 
3840  myDebug->tprintf( 4, "Unpacking tag %s\n", tag_name.c_str() );
3841 
3842  // Create the tag
3843  if( tag_size == MB_VARIABLE_LENGTH )
3844  result = mbImpl->tag_get_handle( tag_name.c_str(), def_val_size, (DataType)tag_data_type, tag_handle,
3845  MB_TAG_VARLEN | MB_TAG_CREAT | MB_TAG_BYTES | tag_type, def_val_ptr );
3846  else
3847  result = mbImpl->tag_get_handle( tag_name.c_str(), tag_size, (DataType)tag_data_type, tag_handle,
3848  MB_TAG_CREAT | MB_TAG_BYTES | tag_type, def_val_ptr );
3849  if( MB_SUCCESS != result ) return result;
3850 
3851  // Get handles and convert to local handles
3852  int num_ents;
3853  UNPACK_INT( buff_ptr, num_ents );
3854  std::vector< EntityHandle > dum_ents( num_ents );
3855  UNPACK_EH( buff_ptr, dum_ents.data(), num_ents );
3856 
3857  // In this case handles are indices into new entity range; need to convert
3858  // to local handles
3859  result = get_local_handles( dum_ents.data(), num_ents, entities );
3860  MB_CHK_SET_ERR( result, "Unable to convert to local handles" );
3861 
3862  // If it's a handle type, also convert tag vals in-place in buffer
3863  if( MB_TYPE_HANDLE == tag_type )
3864  {
3865  dum_ehvals.resize( num_ents );
3866  UNPACK_EH( buff_ptr, dum_ehvals.data(), num_ents );
3867  result = get_local_handles( dum_ehvals.data(), num_ents, entities );
3868  MB_CHK_SET_ERR( result, "Failed to get local handles for tag vals" );
3869  }
3870 
3871  DataType data_type;
3872  mbImpl->tag_get_data_type( tag_handle, data_type );
3873  int type_size = TagInfo::size_from_data_type( data_type );
3874 
3875  if( !dum_ents.empty() )
3876  {
3877  if( tag_size == MB_VARIABLE_LENGTH )
3878  {
3879  // Be careful of alignment here. If the integers are aligned
3880  // in the buffer, we can use them directly. Otherwise we must
3881  // copy them.
3882  std::vector< int > var_lengths( num_ents );
3883  UNPACK_INTS( buff_ptr, var_lengths.data(), num_ents );
3884  UPC( sizeof( int ) * num_ents, " void" );
3885 
3886  // Get pointers into buffer for each tag value
3887  var_len_vals.resize( num_ents );
3888  for( std::vector< EntityHandle >::size_type j = 0; j < (std::vector< EntityHandle >::size_type)num_ents;
3889  j++ )
3890  {
3891  var_len_vals[j] = buff_ptr;
3892  buff_ptr += var_lengths[j] * type_size;
3893  UPC( var_lengths[j], " void" );
3894  }
3895  result = mbImpl->tag_set_by_ptr( tag_handle, dum_ents.data(), num_ents, var_len_vals.data(),
3896  var_lengths.data() );
3897  MB_CHK_SET_ERR( result, "Failed to set tag data when unpacking variable-length tag" );
3898  }
3899  else
3900  {
3901  // Get existing values of dst tag
3902  dum_vals.resize( tag_size * num_ents );
3903  if( mpi_op )
3904  {
3905  int tag_length;
3906  result = mbImpl->tag_get_length( tag_handle, tag_length );
3907  MB_CHK_SET_ERR( result, "Failed to get tag length" );
3908  result = mbImpl->tag_get_data( tag_handle, dum_ents.data(), num_ents, dum_vals.data() );
3909  MB_CHK_SET_ERR( result, "Failed to get existing value of dst tag on entities" );
3910  result = reduce_void( tag_data_type, *mpi_op, tag_length * num_ents, dum_vals.data(), buff_ptr );
3911  MB_CHK_SET_ERR( result, "Failed to perform mpi op on dst tags" );
3912  }
3913  result = mbImpl->tag_set_data( tag_handle, dum_ents.data(), num_ents, buff_ptr );
3914  MB_CHK_SET_ERR( result, "Failed to set range-based tag data when unpacking tag" );
3915  buff_ptr += num_ents * tag_size;
3916  UPC( num_ents * tag_size, " void" );
3917  }
3918  }
3919  }
3920 
3921  myDebug->tprintf( 4, "Done unpacking tags.\n" );
3922 
3923  return MB_SUCCESS;
3924 }

References ErrorCode, get_local_handles(), MB_CHK_SET_ERR, MB_SUCCESS, MB_TAG_BYTES, MB_TAG_CREAT, MB_TAG_VARLEN, MB_TYPE_HANDLE, MB_VARIABLE_LENGTH, mbImpl, myDebug, reduce_void(), moab::TagInfo::size_from_data_type(), moab::Interface::tag_get_data(), moab::Interface::tag_get_data_type(), moab::Interface::tag_get_handle(), moab::Interface::tag_get_length(), moab::Interface::tag_set_by_ptr(), moab::Interface::tag_set_data(), moab::DebugOutput::tprintf(), moab::UNPACK_EH(), moab::UNPACK_INT(), moab::UNPACK_INTS(), and UPC.

Referenced by exchange_tags(), reduce_tags(), and unpack_buffer().

◆ update_iface_sets()

ErrorCode moab::ParallelComm::update_iface_sets ( Range &  sent_ents,
std::vector< EntityHandle > &  remote_handles,
int  from_proc 
)
private

for any remote_handles set to zero, remove corresponding sent_ents from iface_sets corresponding to from_proc

◆ update_remote_data() [1/3]

ErrorCode moab::ParallelComm::update_remote_data ( const EntityHandle  new_h,
const int *  ps,
const EntityHandle *  hs,
const int  num_ps,
const unsigned char  add_pstat 
)
private

Definition at line 2717 of file ParallelComm.cpp.

2729 {
2730  // Get initial sharing data; tag_ps and tag_hs get terminated with -1 and 0
2731  // in this function, so no need to initialize; sharing data does not include
2732  // this proc if shared with only one other
2733 
2734  // Following variables declared here to avoid compiler errors
2735  int new_numps;
2736  unsigned char new_pstat;
2737  std::vector< int > new_ps( MAX_SHARING_PROCS, -1 );
2738  std::vector< EntityHandle > new_hs( MAX_SHARING_PROCS, 0 );
2739 
2740  new_numps = 0;
2741  ErrorCode result = get_sharing_data( new_h, new_ps.data(), new_hs.data(), new_pstat, new_numps );
2742  MB_CHK_SET_ERR( result, "Failed to get sharing data in update_remote_data" );
2743  int num_exist = new_numps;
2744 
2745  // Add new pstat info to the flag
2746  new_pstat |= add_pstat;
2747 
2748  /*
2749  #define plist(str, lst, siz) \
2750  std::cout << str << "("; \
2751  for (int i = 0; i < (int)siz; i++) std::cout << lst[i] << " "; \
2752  std::cout << ") "; \
2753 
2754  std::cout << "update_remote_data: rank = " << rank() << ", new_h = " << new_h << std::endl;
2755  std::string ostr;
2756  plist("ps", ps, num_ps);
2757  plist("hs", hs, num_ps);
2758  print_pstatus(add_pstat, ostr);
2759  std::cout << ", add_pstat = " << ostr.c_str() << std::endl;
2760  plist("tag_ps", new_ps, new_numps);
2761  plist("tag_hs", new_hs, new_numps);
2762  assert(new_numps <= size());
2763  print_pstatus(new_pstat, ostr);
2764  std::cout << ", tag_pstat=" << ostr.c_str() << std::endl;
2765  */
2766 
2767 #ifndef NDEBUG
2768  {
2769  // Check for duplicates in proc list
2770  std::set< unsigned int > dumprocs;
2771  unsigned int dp = 0;
2772  for( ; (int)dp < num_ps && -1 != ps[dp]; dp++ )
2773  dumprocs.insert( ps[dp] );
2774  assert( dp == dumprocs.size() );
2775  }
2776 #endif
2777 
2778  // If only one sharer and I'm the owner, insert myself in the list;
2779  // otherwise, my data is checked at the end
2780  if( 1 == new_numps && !( new_pstat & PSTATUS_NOT_OWNED ) )
2781  {
2782  new_hs[1] = new_hs[0];
2783  new_ps[1] = new_ps[0];
2784  new_hs[0] = new_h;
2785  new_ps[0] = rank();
2786  new_numps = 2;
2787  }
2788 
2789  // Now put passed-in data onto lists
2790  int idx;
2791  for( int i = 0; i < num_ps; i++ )
2792  {
2793  idx = std::find( new_ps.data(), new_ps.data() + new_numps, ps[i] ) - new_ps.data();
2794  if( idx < new_numps )
2795  {
2796  if( !new_hs[idx] && hs[i] )
2797  // h on list is 0 and passed-in h is non-zero, replace it
2798  new_hs[idx] = hs[i];
2799  else
2800  assert( !hs[i] || new_hs[idx] == hs[i] );
2801  }
2802  else
2803  {
2804  if( new_numps + 1 == MAX_SHARING_PROCS )
2805  {
2806  MB_SET_ERR( MB_FAILURE, "Exceeded MAX_SHARING_PROCS for "
2807  << CN::EntityTypeName( TYPE_FROM_HANDLE( new_h ) ) << ' '
2808  << ID_FROM_HANDLE( new_h ) << " in process " << rank() );
2809  }
2810  new_ps[new_numps] = ps[i];
2811  new_hs[new_numps] = hs[i];
2812  new_numps++;
2813  }
2814  }
2815 
2816  // Add myself, if it isn't there already
2817  idx = std::find( new_ps.data(), new_ps.data() + new_numps, rank() ) - new_ps.data();
2818  if( idx == new_numps )
2819  {
2820  new_ps[new_numps] = rank();
2821  new_hs[new_numps] = new_h;
2822  new_numps++;
2823  }
2824  else if( !new_hs[idx] && new_numps > 2 )
2825  new_hs[idx] = new_h;
2826 
2827  // Proc list is complete; update for shared, multishared
2828  if( new_numps > 1 )
2829  {
2830  if( new_numps > 2 ) new_pstat |= PSTATUS_MULTISHARED;
2831  new_pstat |= PSTATUS_SHARED;
2832  }
2833 
2834  /*
2835  plist("new_ps", new_ps, new_numps);
2836  plist("new_hs", new_hs, new_numps);
2837  print_pstatus(new_pstat, ostr);
2838  std::cout << ", new_pstat=" << ostr.c_str() << std::endl;
2839  std::cout << std::endl;
2840  */
2841 
2842  result = set_sharing_data( new_h, new_pstat, num_exist, new_numps, new_ps.data(), new_hs.data() );
2843  MB_CHK_SET_ERR( result, "Failed to set sharing data in update_remote_data" );
2844 
2845  if( new_pstat & PSTATUS_SHARED ) sharedEnts.insert( new_h );
2846 
2847  return MB_SUCCESS;
2848 }

References moab::CN::EntityTypeName(), ErrorCode, get_sharing_data(), moab::ID_FROM_HANDLE(), MAX_SHARING_PROCS, MB_CHK_SET_ERR, MB_SET_ERR, MB_SUCCESS, PSTATUS_MULTISHARED, PSTATUS_NOT_OWNED, PSTATUS_SHARED, rank(), set_sharing_data(), sharedEnts, and moab::TYPE_FROM_HANDLE().

◆ update_remote_data() [2/3]

ErrorCode moab::ParallelComm::update_remote_data ( EntityHandle  entity,
std::vector< int > &  procs,
std::vector< EntityHandle > &  handles 
)

Definition at line 1035 of file ParallelComm.cpp.

1038 {
1039  ErrorCode error;
1040  unsigned char pstatus = PSTATUS_INTERFACE;
1041 
1042  int procmin = *std::min_element( procs.begin(), procs.end() );
1043 
1044  if( (int)rank() > procmin )
1045  pstatus |= PSTATUS_NOT_OWNED;
1046  else
1047  procmin = rank();
1048 
1049  // DBG
1050  // std::cout<<"entity = "<<entity<<std::endl;
1051  // for (int j=0; j<procs.size(); j++)
1052  // std::cout<<"procs["<<j<<"] = "<<procs[j]<<", handles["<<j<<"] = "<<handles[j]<<std::endl;
1053  // DBG
1054 
1055  if( (int)procs.size() > 1 )
1056  {
1057  procs.push_back( rank() );
1058  handles.push_back( entity );
1059 
1060  int idx = std::find( procs.begin(), procs.end(), procmin ) - procs.begin();
1061 
1062  std::iter_swap( procs.begin(), procs.begin() + idx );
1063  std::iter_swap( handles.begin(), handles.begin() + idx );
1064 
1065  // DBG
1066  // std::cout<<"entity = "<<entity<<std::endl;
1067  // for (int j=0; j<procs.size(); j++)
1068  // std::cout<<"procs["<<j<<"] = "<<procs[j]<<", handles["<<j<<"] = "<<handles[j]<<std::endl;
1069  // DBG
1070  }
1071 
1072  // if ((entity == 10388) && (rank()==1))
1073  // std::cout<<"Here"<<std::endl;
1074 
1075  error = update_remote_data( entity, procs.data(), handles.data(), procs.size(), pstatus );
1076  MB_CHK_ERR( error );
1077 
1078  return MB_SUCCESS;
1079 }

References moab::error(), ErrorCode, MB_CHK_ERR, MB_SUCCESS, PSTATUS_INTERFACE, PSTATUS_NOT_OWNED, and rank().

Referenced by resolve_shared_ents(), unpack_entities(), unpack_remote_handles(), unpack_sets(), and update_remote_data().

◆ update_remote_data() [3/3]

ErrorCode moab::ParallelComm::update_remote_data ( Range &  local_range,
Range &  remote_range,
int  other_proc,
const unsigned char  add_pstat 
)
private

Definition at line 2699 of file ParallelComm.cpp.

2703 {
2704  Range::iterator rit, rit2;
2705  ErrorCode result = MB_SUCCESS;
2706 
2707  // For each pair of local/remote handles:
2708  for( rit = local_range.begin(), rit2 = remote_range.begin(); rit != local_range.end(); ++rit, ++rit2 )
2709  {
2710  result = update_remote_data( *rit, &other_proc, &( *rit2 ), 1, add_pstat );
2711  MB_CHK_ERR( result );
2712  }
2713 
2714  return MB_SUCCESS;
2715 }

References moab::Range::begin(), moab::Range::end(), ErrorCode, MB_CHK_ERR, MB_SUCCESS, and update_remote_data().

◆ update_remote_data_old()

ErrorCode moab::ParallelComm::update_remote_data_old ( const EntityHandle  new_h,
const int *  ps,
const EntityHandle *  hs,
const int  num_ps,
const unsigned char  add_pstat 
)
private

Definition at line 2850 of file ParallelComm.cpp.

2855 {
2856  EntityHandle tag_hs[MAX_SHARING_PROCS];
2857  int tag_ps[MAX_SHARING_PROCS];
2858  unsigned char pstat;
2859  // Get initial sharing data; tag_ps and tag_hs get terminated with -1 and 0
2860  // in this function, so no need to initialize
2861  unsigned int num_exist;
2862  ErrorCode result = get_sharing_data( new_h, tag_ps, tag_hs, pstat, num_exist );
2863  MB_CHK_ERR( result );
2864 
2865 #ifndef NDEBUG
2866  {
2867  // Check for duplicates in proc list
2868  std::set< unsigned int > dumprocs;
2869  unsigned int dp = 0;
2870  for( ; (int)dp < num_ps && -1 != ps[dp]; dp++ )
2871  dumprocs.insert( ps[dp] );
2872  assert( dp == dumprocs.size() );
2873  }
2874 #endif
2875 
2876  // Add any new sharing data
2877  bool changed = false;
2878  int idx;
2879  if( !num_exist )
2880  {
2881  // Just take what caller passed
2882  memcpy( tag_ps, ps, num_ps * sizeof( int ) );
2883  memcpy( tag_hs, hs, num_ps * sizeof( EntityHandle ) );
2884  num_exist = num_ps;
2885  // If it's only one, hopefully I'm not there yet...
2886  assert( "I shouldn't be the only proc there." && ( 1 != num_exist || ps[0] != (int)procConfig.proc_rank() ) );
2887  changed = true;
2888  }
2889  else
2890  {
2891  for( int i = 0; i < num_ps; i++ )
2892  {
2893  idx = std::find( tag_ps, tag_ps + num_exist, ps[i] ) - tag_ps;
2894  if( idx == (int)num_exist )
2895  {
2896  if( num_exist == MAX_SHARING_PROCS )
2897  {
2898  std::cerr << "Exceeded MAX_SHARING_PROCS for " << CN::EntityTypeName( TYPE_FROM_HANDLE( new_h ) )
2899  << ' ' << ID_FROM_HANDLE( new_h ) << " in process " << proc_config().proc_rank()
2900  << std::endl;
2901  std::cerr.flush();
2902  MPI_Abort( proc_config().proc_comm(), 66 );
2903  }
2904 
2905  // If there's only 1 sharing proc, and it's not me, then
2906  // we'll end up with 3; add me to the front
2907  if( !i && num_ps == 1 && num_exist == 1 && ps[0] != (int)procConfig.proc_rank() )
2908  {
2909  int j = 1;
2910  // If I own this entity, put me at front, otherwise after first
2911  if( !( pstat & PSTATUS_NOT_OWNED ) )
2912  {
2913  tag_ps[1] = tag_ps[0];
2914  tag_hs[1] = tag_hs[0];
2915  j = 0;
2916  }
2917  tag_ps[j] = procConfig.proc_rank();
2918  tag_hs[j] = new_h;
2919  num_exist++;
2920  }
2921 
2922  tag_ps[num_exist] = ps[i];
2923  tag_hs[num_exist] = hs[i];
2924  num_exist++;
2925  changed = true;
2926  }
2927  else if( 0 == tag_hs[idx] )
2928  {
2929  tag_hs[idx] = hs[i];
2930  changed = true;
2931  }
2932  else if( 0 != hs[i] )
2933  {
2934  assert( hs[i] == tag_hs[idx] );
2935  }
2936  }
2937  }
2938 
2939  // Adjust for interface layer if necessary
2940  if( add_pstat & PSTATUS_INTERFACE )
2941  {
2942  idx = std::min_element( tag_ps, tag_ps + num_exist ) - tag_ps;
2943  if( idx )
2944  {
2945  int tag_proc = tag_ps[idx];
2946  tag_ps[idx] = tag_ps[0];
2947  tag_ps[0] = tag_proc;
2948  EntityHandle tag_h = tag_hs[idx];
2949  tag_hs[idx] = tag_hs[0];
2950  tag_hs[0] = tag_h;
2951  changed = true;
2952  if( tag_ps[0] != (int)procConfig.proc_rank() ) pstat |= PSTATUS_NOT_OWNED;
2953  }
2954  }
2955 
2956  if( !changed ) return MB_SUCCESS;
2957 
2958  assert( "interface entities should have > 1 proc" && ( !( add_pstat & PSTATUS_INTERFACE ) || num_exist > 1 ) );
2959  assert( "ghost entities should have > 1 proc" && ( !( add_pstat & PSTATUS_GHOST ) || num_exist > 1 ) );
2960 
2961  // If it's multi-shared and we created the entity in this unpack,
2962  // local handle probably isn't in handle list yet
2963  if( num_exist > 2 )
2964  {
2965  idx = std::find( tag_ps, tag_ps + num_exist, procConfig.proc_rank() ) - tag_ps;
2966  assert( idx < (int)num_exist );
2967  if( !tag_hs[idx] ) tag_hs[idx] = new_h;
2968  }
2969 
2970  int tag_p;
2971  EntityHandle tag_h;
2972 
2973  // Update pstat
2974  pstat |= add_pstat;
2975 
2976  if( num_exist > 2 )
2977  pstat |= ( PSTATUS_MULTISHARED | PSTATUS_SHARED );
2978  else if( num_exist > 0 )
2979  pstat |= PSTATUS_SHARED;
2980 
2981  // compare_remote_data(new_h, num_ps, hs, ps, add_pstat,
2982  // num_exist, tag_hs, tag_ps, pstat);
2983 
2984  // Reset single shared proc/handle if was shared and moving to multi-shared
2985  if( num_exist > 2 && !( pstat & PSTATUS_MULTISHARED ) && ( pstat & PSTATUS_SHARED ) )
2986  {
2987  // Must remove sharedp/h first, which really means set to default value
2988  tag_p = -1;
2989  result = mbImpl->tag_set_data( sharedp_tag(), &new_h, 1, &tag_p );
2990  MB_CHK_SET_ERR( result, "Failed to set sharedp tag data" );
2991  tag_h = 0;
2992  result = mbImpl->tag_set_data( sharedh_tag(), &new_h, 1, &tag_h );
2993  MB_CHK_SET_ERR( result, "Failed to set sharedh tag data" );
2994  }
2995 
2996  // Set sharing tags
2997  if( num_exist > 2 )
2998  {
2999  std::fill( tag_ps + num_exist, tag_ps + MAX_SHARING_PROCS, -1 );
3000  std::fill( tag_hs + num_exist, tag_hs + MAX_SHARING_PROCS, 0 );
3001  result = mbImpl->tag_set_data( sharedps_tag(), &new_h, 1, tag_ps );
3002  MB_CHK_SET_ERR( result, "Failed to set sharedps tag data" );
3003  result = mbImpl->tag_set_data( sharedhs_tag(), &new_h, 1, tag_hs );
3004  MB_CHK_SET_ERR( result, "Failed to set sharedhs tag data" );
3005 
3006 #ifndef NDEBUG
3007  {
3008  // Check for duplicates in proc list
3009  std::set< unsigned int > dumprocs;
3010  unsigned int dp = 0;
3011  for( ; dp < num_exist && -1 != tag_ps[dp]; dp++ )
3012  dumprocs.insert( tag_ps[dp] );
3013  assert( dp == dumprocs.size() );
3014  }
3015 #endif
3016  }
3017  else if( num_exist == 2 || num_exist == 1 )
3018  {
3019  if( tag_ps[0] == (int)procConfig.proc_rank() )
3020  {
3021  assert( 2 == num_exist && tag_ps[1] != (int)procConfig.proc_rank() );
3022  tag_ps[0] = tag_ps[1];
3023  tag_hs[0] = tag_hs[1];
3024  }
3025  assert( tag_ps[0] != -1 && tag_hs[0] != 0 );
3026  result = mbImpl->tag_set_data( sharedp_tag(), &new_h, 1, tag_ps );
3027  MB_CHK_SET_ERR( result, "Failed to set sharedp tag data" );
3028  result = mbImpl->tag_set_data( sharedh_tag(), &new_h, 1, tag_hs );
3029  MB_CHK_SET_ERR( result, "Failed to set sharedh tag data" );
3030  }
3031 
3032  // Now set new pstatus
3033  result = mbImpl->tag_set_data( pstatus_tag(), &new_h, 1, &pstat );
3034  MB_CHK_SET_ERR( result, "Failed to set pstatus tag data" );
3035 
3036  if( pstat & PSTATUS_SHARED ) sharedEnts.insert( new_h );
3037 
3038  return MB_SUCCESS;
3039 }

References moab::CN::EntityTypeName(), ErrorCode, get_sharing_data(), moab::ID_FROM_HANDLE(), MAX_SHARING_PROCS, MB_CHK_ERR, MB_CHK_SET_ERR, MB_SUCCESS, mbImpl, proc_config(), moab::ProcConfig::proc_rank(), procConfig, PSTATUS_GHOST, PSTATUS_INTERFACE, PSTATUS_MULTISHARED, PSTATUS_NOT_OWNED, PSTATUS_SHARED, pstatus_tag(), sharedEnts, sharedh_tag(), sharedhs_tag(), sharedp_tag(), sharedps_tag(), moab::Interface::tag_set_data(), and moab::TYPE_FROM_HANDLE().

Friends And Related Function Documentation

◆ ParallelMergeMesh

friend class ParallelMergeMesh
friend

Definition at line 57 of file ParallelComm.hpp.

Member Data Documentation

◆ ackbuff

int moab::ParallelComm::ackbuff
private

Definition at line 1475 of file ParallelComm.hpp.

Referenced by recv_entities(), and send_entities().

◆ buffProcs

std::vector< unsigned int > moab::ParallelComm::buffProcs
private

processor rank for each buffer index

Definition at line 1456 of file ParallelComm.hpp.

Referenced by buff_procs(), check_all_shared_handles(), check_my_shared_handles(), correct_thin_ghost_layers(), exchange_all_shared_handles(), exchange_ghost_cells(), exchange_owned_mesh(), exchange_tags(), get_buffers(), get_comm_procs(), get_sent_ents(), initialize(), pack_shared_handles(), post_irecv(), recv_entities(), recv_messages(), recv_remote_handle_messages(), reduce_tags(), send_entities(), send_recv_entities(), and settle_intersection_points().

◆ errorHandler

Error* moab::ParallelComm::errorHandler
private

Error handler.

Definition at line 1438 of file ParallelComm.hpp.

Referenced by initialize(), and packed_tag_size().

◆ globalPartCount

int moab::ParallelComm::globalPartCount
private

Cache of global part count.

Definition at line 1467 of file ParallelComm.hpp.

Referenced by collective_sync_partition(), create_part(), destroy_part(), and get_global_part_count().

◆ ifaceSetsTag

Tag moab::ParallelComm::ifaceSetsTag
private

Definition at line 1465 of file ParallelComm.hpp.

◆ INITIAL_BUFF_SIZE

const unsigned int moab::ParallelComm::INITIAL_BUFF_SIZE = 1024
static

Definition at line 951 of file ParallelComm.hpp.

Referenced by broadcast_entities(), exchange_ghost_cells(), exchange_owned_mesh(), exchange_tags(), get_buffers(), post_irecv(), recv_buffer(), recv_entities(), reduce_tags(), moab::ParallelComm::Buffer::reset_buffer(), scatter_entities(), send_buffer(), moab::ParCommGraph::send_mesh_parts(), send_recv_entities(), and settle_intersection_points().

◆ interfaceSets

Range moab::ParallelComm::interfaceSets
private

Definition at line 1459 of file ParallelComm.hpp.

Referenced by check_clean_iface(), create_iface_pc_links(), create_interface_sets(), get_ghosted_entities(), get_iface_entities(), get_interface_procs(), interface_sets(), and tag_iface_entities().

◆ localOwnedBuffs

std::vector< Buffer* > moab::ParallelComm::localOwnedBuffs
private

more data buffers, proc-specific

Definition at line 1441 of file ParallelComm.hpp.

Referenced by delete_all_buffers(), exchange_ghost_cells(), exchange_owned_mesh(), exchange_tags(), get_buffers(), initialize(), post_irecv(), recv_entities(), recv_messages(), recv_remote_handle_messages(), reduce_tags(), reset_all_buffers(), send_entities(), send_recv_entities(), and settle_intersection_points().

◆ mbImpl

Interface* moab::ParallelComm::mbImpl
private

MB interface associated with this writer.

Definition at line 1429 of file ParallelComm.hpp.

Referenced by add_pcomm(), add_verts(), assign_entities_part(), assign_global_ids(), augment_default_sets_with_ghosts(), check_all_shared_handles(), check_clean_iface(), check_global_ids(), check_local_shared(), check_sent_ents(), clean_shared_tags(), correct_thin_ghost_layers(), create_iface_pc_links(), create_interface_sets(), create_part(), delete_entities(), destroy_part(), estimate_ents_buffer_size(), estimate_sets_buffer_size(), exchange_ghost_cells(), exchange_owned_meshs(), exchange_tags(), filter_pstatus(), find_existing_entity(), gather_data(), get_ghosted_entities(), get_iface_entities(), get_interface_procs(), get_moab(), get_owner_handle(), get_owning_part(), get_part_entities(), get_proc_nvecs(), get_pstatus(), get_pstatus_entities(), get_remote_handles(), get_sharing_data(), get_sharing_parts(), get_tag_send_list(), initialize(), is_iface_proc(), list_entities(), pack_entities(), pack_entity_seq(), pack_sets(), pack_tag(), partition_tag(), pstatus_tag(), reduce_tags(), remove_entities_part(), remove_pcomm(), resolve_shared_ents(), resolve_shared_sets(), set_partitioning(), set_pstatus_entities(), set_sharing_data(), settle_intersection_points(), sharedh_tag(), sharedhs_tag(), sharedp_tag(), sharedps_tag(), tag_iface_entities(), tag_shared_verts(), unpack_entities(), unpack_remote_handles(), unpack_sets(), unpack_tags(), and update_remote_data_old().

◆ myDebug

DebugOutput* moab::ParallelComm::myDebug
private

used to set verbosity level and to report output

Definition at line 1478 of file ParallelComm.hpp.

Referenced by correct_thin_ghost_layers(), define_mpe(), exchange_ghost_cells(), exchange_owned_mesh(), exchange_tags(), get_debug_verbosity(), initialize(), pack_entities(), pack_entity_seq(), pack_sets(), pack_tag(), pack_tags(), print_debug_irecv(), print_debug_isend(), print_debug_recd(), print_debug_waitany(), recv_entities(), reduce_tags(), resolve_shared_ents(), send_entities(), send_recv_entities(), set_debug_verbosity(), settle_intersection_points(), unpack_buffer(), unpack_entities(), unpack_sets(), unpack_tags(), and ~ParallelComm().

◆ myFile

std::ofstream moab::ParallelComm::myFile
private

Definition at line 1471 of file ParallelComm.hpp.

◆ partitioningSet

EntityHandle moab::ParallelComm::partitioningSet
private

entity set containing all parts

Definition at line 1469 of file ParallelComm.hpp.

Referenced by get_partitioning(), and set_partitioning().

◆ partitionSets

Range moab::ParallelComm::partitionSets
private

the partition, interface sets for this comm'n instance

Definition at line 1459 of file ParallelComm.hpp.

Referenced by get_part_entities(), partition_sets(), resolve_shared_ents(), and moab::ParallelMergeMesh::TagSharedElements().

◆ partitionTag

Tag moab::ParallelComm::partitionTag
private

Definition at line 1465 of file ParallelComm.hpp.

Referenced by partition_tag().

◆ pcommID

int moab::ParallelComm::pcommID
private

Definition at line 1473 of file ParallelComm.hpp.

Referenced by get_id(), initialize(), and ParallelComm().

◆ PROC_OWNER

unsigned char moab::ParallelComm::PROC_OWNER
static

Definition at line 88 of file ParallelComm.hpp.

◆ PROC_SHARED

unsigned char moab::ParallelComm::PROC_SHARED
static

Definition at line 88 of file ParallelComm.hpp.

◆ procConfig

ProcConfig moab::ParallelComm::procConfig
private

Proc config object, keeps info on parallel stuff.

Definition at line 1432 of file ParallelComm.hpp.

Referenced by assign_global_ids(), augment_default_sets_with_ghosts(), broadcast_entities(), build_sharedhps_list(), check_all_shared_handles(), check_clean_iface(), check_local_shared(), create_interface_sets(), delete_entities(), exchange_all_shared_handles(), exchange_ghost_cells(), exchange_owned_mesh(), exchange_tags(), get_buffers(), get_interface_procs(), get_proc_nvecs(), get_remote_handles(), initialize(), ParallelComm(), post_irecv(), print_buffer(), proc_config(), recv_buffer(), recv_entities(), recv_messages(), recv_remote_handle_messages(), reduce_tags(), resolve_shared_ents(), resolve_shared_sets(), scatter_entities(), send_buffer(), send_entities(), send_recv_entities(), set_rank(), set_sharing_data(), set_size(), settle_intersection_points(), tag_shared_verts(), unpack_entities(), and update_remote_data_old().

◆ pstatusTag

Tag moab::ParallelComm::pstatusTag
private

Definition at line 1465 of file ParallelComm.hpp.

Referenced by pstatus_tag().

◆ recvRemotehReqs

std::vector< MPI_Request > moab::ParallelComm::recvRemotehReqs
private

Definition at line 1453 of file ParallelComm.hpp.

Referenced by exchange_owned_meshs(), post_irecv(), recv_entities(), and send_entities().

◆ recvReqs

std::vector< MPI_Request > moab::ParallelComm::recvReqs
private

receive request objects

Definition at line 1453 of file ParallelComm.hpp.

Referenced by exchange_owned_meshs(), post_irecv(), recv_entities(), recv_messages(), send_entities(), and set_recv_request().

◆ remoteOwnedBuffs

std::vector< Buffer* > moab::ParallelComm::remoteOwnedBuffs
private

Definition at line 1441 of file ParallelComm.hpp.

Referenced by delete_all_buffers(), exchange_ghost_cells(), exchange_owned_mesh(), exchange_tags(), get_buffers(), initialize(), post_irecv(), recv_entities(), recv_messages(), recv_remote_handle_messages(), reduce_tags(), reset_all_buffers(), send_entities(), send_recv_entities(), and settle_intersection_points().

◆ sendReqs

std::vector< MPI_Request > moab::ParallelComm::sendReqs
private

request objects, may be used if store_remote_handles is used

Definition at line 1450 of file ParallelComm.hpp.

Referenced by exchange_ghost_cells(), exchange_owned_mesh(), exchange_tags(), post_irecv(), recv_entities(), recv_messages(), recv_remote_handle_messages(), reduce_tags(), send_entities(), send_recv_entities(), set_send_request(), and settle_intersection_points().

◆ sequenceManager

SequenceManager* moab::ParallelComm::sequenceManager
private

Sequence manager, to get more efficient access to entities.

Definition at line 1435 of file ParallelComm.hpp.

Referenced by get_tag_send_list(), initialize(), pack_entities(), and packed_tag_size().

◆ sharedEnts

std::set< EntityHandle > moab::ParallelComm::sharedEnts
private

all local entities shared with others, whether ghost or ghosted

Definition at line 1462 of file ParallelComm.hpp.

Referenced by augment_default_sets_with_ghosts(), check_local_shared(), check_my_shared_handles(), correct_thin_ghost_layers(), create_interface_sets(), delete_entities(), exchange_ghost_cells(), exchange_owned_meshs(), exchange_tags(), get_shared_entities(), list_entities(), pack_shared_handles(), reduce_tags(), resolve_shared_ents(), set_sharing_data(), tag_shared_verts(), update_remote_data(), and update_remote_data_old().

◆ sharedhsTag

Tag moab::ParallelComm::sharedhsTag
private

Definition at line 1465 of file ParallelComm.hpp.

Referenced by sharedhs_tag().

◆ sharedhTag

Tag moab::ParallelComm::sharedhTag
private

Definition at line 1465 of file ParallelComm.hpp.

Referenced by sharedh_tag().

◆ sharedpsTag

Tag moab::ParallelComm::sharedpsTag
private

Definition at line 1465 of file ParallelComm.hpp.

Referenced by sharedps_tag().

◆ sharedpTag

Tag moab::ParallelComm::sharedpTag
private

tags used to save sharing procs and handles

Definition at line 1465 of file ParallelComm.hpp.

Referenced by sharedp_tag().

◆ sharedSetData

SharedSetData* moab::ParallelComm::sharedSetData
private

Data about shared sets.

Definition at line 1481 of file ParallelComm.hpp.

Referenced by get_entityset_local_handle(), get_entityset_owner(), get_entityset_owners(), get_entityset_procs(), get_owned_sets(), get_shared_sets(), ParallelComm(), resolve_shared_sets(), and ~ParallelComm().

The documentation for this class was generated from the following files: