Basic Object Design and Implementation

PETSc is designed by using strong data encapsulation. Hence, any collection of data (for instance, a sparse matrix) is stored in a way that is completely private from the application code. The application code can manipulate the data only through a well-defined interface, since it does not “know” how the data is stored internally.

Introduction

PETSc is designed around several classes including Vec (vectors) and Mat (matrices, both dense and sparse). Each class is implemented by using a C struct that contains the data and function pointers for operations on the data (much like virtual functions in C++ classes). Each class consists of three parts:

A (small) common part shared by all PETSc classes (for example, both KSP and PC have this same header).

Another common part shared by all PETSc implementations of the class (for example, both KSPGMRES and KSPCG have this common subheader).

A private part used by only one particular implementation written in PETSc.

For example, all matrix (Mat) classes share a function table of operations that may be performed on the matrix; all PETSc matrix implementations share some additional data fields, including matrix parallel layout, while a particular matrix implementation in PETSc (say compressed sparse row) has its own data fields for storing the actual matrix values and sparsity pattern. This will be explained in more detail in the following sections. New class implementations must use the PETSc common part.

We will use <class>_<implementation> to denote the actual source code and data structures used for a particular implementation of an object that has the <class> interface.

Organization of the Source Code

Each class has the following organization.

Its own, application-public, include file include/petsc<class>.h.

Its own directory, src/<class> or src/<package>/<class>.

A data structure defined in the file include/petsc/private/<class>impl.h. This data structure is shared by all the different PETSc implementations of the class. For example, for matrices it is shared by dense, sparse, parallel, and sequential formats.

An abstract interface that defines the application-callable functions for the class. These are defined in the directory src/<class>/interface. This is how polymorphism is supported with code that implements the abstract interface to the operations on the object. Essentially, these routines do some error checking of arguments and logging of profiling information and then call the function appropriate for the particular implementation of the object. The name of the abstract function is <class>Operation, for instance, MatMult() or PCCreate(), while the name of a particular implementation is <class>Operation_<implementation>, for instance, MatMult_SeqAIJ() or PCCreate_ILU(). These naming conventions are used to simplify code maintenance (also see Section [sec:stylenames]).

One or more actual implementations of the class (for example, sparse uniprocessor and parallel matrices implemented with the AIJ storage format). These are each in a subdirectory of src/<class>/impls. Except in rare circumstances, data structures defined here should not be referenced from outside this directory.

Each type of object (for instance, a vector) is defined in its own public include file, by typedef _p_<class>* <class>; (for example, typedef _p_Vec* Vec;). This organization allows the compiler to perform type checking on all subroutine calls while at the same time completely removing the details of the implementation of _p_<class> from the application code. This capability is extremely important because it allows the library internals to be changed without altering or recompiling the application code.

Common Object Header

All PETSc/PETSc objects have the following common header structures defined in include/petsc/private/petscimpl.h:

typedef struct {
  PetscErrorCode (*getcomm)(PetscObject,MPI_Comm*);
  PetscErrorCode (*view)(PetscObject,Viewer);
  PetscErrorCode (*destroy)(PetscObject);
  PetscErrorCode (*query)(PetscObject,const char*,PetscObject*);
  PetscErrorCode (*compose)(PetscObject,const char*,PetscObject);
  PetscErrorCode (*composefunction)(PetscObject,const char*,void(*)(void));
  PetscErrorCode (*queryfunction)(PetscObject,const char*,void (**)(void));
} PetscOps;

struct _p_<class> {
  PetscClassId     classid;
  PetscOps         *bops;
  <class>Ops       *ops;
  MPI_Comm         comm;
  PetscLogDouble   flops,time,mem;
  int              id;
  int              refct;
  int              tag;
  DLList           qlist;
  OList            olist;
  char             *type_name;
  PetscObject      parent;
  char             *name;
  char             *prefix;
  void             *cpp;
  void             **fortran_func_pointers;
  ..........
  CLASS-SPECIFIC DATASTRUCTURES
};

Here <class>ops is a function table (like the PetscOps above) that contains the function pointers for the operations specific to that class. For example, the PETSc vector class object operations in include/petsc/private/vecimpl.h include the following.

typedef struct _VecOps* VecOps;
struct _VecOps {
  PetscErrorCode (*duplicate)(Vec,Vec*); /* get single vector */
  PetscErrorCode (*duplicatevecs)(Vec,PetscInt,Vec**); /* get array of vectors */
  PetscErrorCode (*destroyvecs)(PetscInt,Vec[]); /* free array of vectors */
  PetscErrorCode (*dot)(Vec,Vec,PetscScalar*); /* z = x^H * y */
  PetscErrorCode (*mdot)(Vec,PetscInt,const Vec[],PetscScalar*); /* z[j] = x dot y[j] */
  PetscErrorCode (*norm)(Vec,NormType,PetscReal*); /* z = sqrt(x^H * x) */
  PetscErrorCode (*tdot)(Vec,Vec,PetscScalar*); /* x'*y */
  PetscErrorCode (*mtdot)(Vec,PetscInt,const Vec[],PetscScalar*);/* z[j] = x dot y[j] */
  PetscErrorCode (*scale)(Vec,PetscScalar);  /* x = alpha * x   */
  PetscErrorCode (*copy)(Vec,Vec); /* y = x */
  PetscErrorCode (*set)(Vec,PetscScalar); /* y = alpha  */
  PetscErrorCode (*swap)(Vec,Vec); /* exchange x and y */
  PetscErrorCode (*axpy)(Vec,PetscScalar,Vec); /* y = y + alpha * x */
  PetscErrorCode (*axpby)(Vec,PetscScalar,PetscScalar,Vec); /* y = alpha * x + beta * y*/
  PetscErrorCode (*maxpy)(Vec,PetscInt,const PetscScalar*,Vec*); /* y = y + alpha[j] x[j] */
  ... (AND SO ON) ...
};

struct _p_Vec {
  PetscClassId           classid;
  PetscOps               *bops;
  VecOps                 *ops;
  MPI_Comm               comm;
  PetscLogDouble         flops,time,mem;
  int                    id;
  int                    refct;
  int                    tag;
  DLList                 qlist;
  OList                  olist;
  char                   *type_name;
  PetscObject            parent;
  char                   *name;
  char                   *prefix;
  void                   **fortran_func_pointers;
  void                   *data;     /* implementation-specific data */
  PetscLayout            map;
  ISLocalToGlobalMapping mapping;   /* mapping used in VecSetValuesLocal() */
};

Each PETSc object begins with a PetscClassId, which is used for error checking. Each different class of objects has its value for classid; these are used to distinguish between classes. When a new class is created you need to call

PetscClassIdRegister(const char *classname,PetscClassId *classid);

For example,

PetscClassIdRegister("index set",&IS_CLASSID);

you can verify that an object is valid of a particular class with PetscValidHeaderSpecific, for example,

PetscValidHeaderSpecific(x,VEC_CLASSID,1);

The third argument to this macro indicates the position in the calling sequence of the function the object was passed in. This is to generate more complete error messages.

To check for an object of any type, use

PetscValidHeader(x,1);

The obj->ops functions provide implementations of the standard methods of the object class. Each type of the class may have different function pointers in the array. Subtypes sometimes replace some of the function pointers of the parent, so they play the role of virtual methods in C++.

PETSc code that calls these function pointers should be done via

PetscUseTypeMethod(obj,method,other arguments);
PetscTryTypeMethod(obj,method,other arguments);

For example,

PetscErrorCode XXXOp(XXX x,YYY y)
{
  PetscFunctionBegin;
  PetscUseTypeMethod(x,op,y);
  PetscFunctionReturn(PETSC_SUCCESS);
}

The Try variant skips the function call if the method has not been set while the Use version generates an error in that case.

See also, PetscUseMethod(), and PetscTryMethod().

Common Object Functions

Several routines are provided for manipulating data within the header. These include the specific functions in the PETSc common function table. The function pointers are not called directly; rather you should call PetscObjectFunctionName(), where FunctionName is one of the functions listed below with the first letter of each word capitalized.

getcomm(PetscObject,MPI_Comm*) obtains the MPI communicator associated with this object.

view(PetscObject,PetscViewer) allows you to store or visualize the data inside an object. If the Viewer is NULL, then it should cause the object to print information on the object to textttstdout.

destroy(PetscObject) causes the reference count of the object to be decreased by one or the object to be destroyed and all memory used by the object to be freed when the reference count drops to zero. If the object has any other objects composed with it, they are each sent a destroy(); that is, the destroy() function is called on them also.

compose(PetscObject,const char *name,PetscObject) associates the second object with the first object and increases the reference count of the second object. If an object with the same name was previously composed, that object is dereferenced and replaced with the new object. If the second object is NULL and an object with the same name has already been composed, that object is dereferenced (the destroy() function is called on it, and that object is removed from the first object). This is a way to remove, by name, an object that was previously composed.

query(PetscObject,const char *name,PetscObject*) retrieves an object that was previously composed with the first object via PetscObjectCompose(). It retrieves a NULL if no object with that name was previously composed.

composefunction(PetscObject,const char *name,void *func) associates a function pointer with an object. If the object already had a composed function with the same name, the old one is replaced. If func is NULL, the existing function is removed from the object. The string name is the character string name of the function.

For example, fname may be PCCreate_LU.

queryfunction(PetscObject,const char *name,void **func) retrieves a function pointer that was associated with the object via PetscObjectComposeFunction(). If dynamic libraries are used, the function is loaded into memory at this time (if it has not been previously loaded), not when the composefunction() routine was called.

Since the object composition allows one to compose PETSc objects only with PETSc objects rather than any arbitrary pointer, PETSc provides the convenience object PetscContainer, created with the routine PetscContainerCreate(MPI_Comm,PetscContainer*), to allow wrapping any kind of data into a PETSc object that can then be composed with a PETSc object.

Object Function Implementation

This section discusses how PETSc implements the compose(), query(), composefunction(), and queryfunction() functions for its object implementations. Other PETSc-compatible class implementations are free to manage these functions in any manner; but unless there is a specific reason, they should use the PETSc defaults so that the library writer does not have to “reinvent the wheel.”

Compose and Query Objects

In src/sys/objects/olist.c, PETSc defines a C struct

typedef struct _PetscObjectList* PetscObjectList;
struct _PetscObjectList {
    char             name[128];
    PetscObject      obj;
    PetscObjectList  next;
};

from which linked lists of composed objects may be constructed. The routines to manipulate these elementary objects are

int PetscObjectListAdd(PetscObjectList *fl,const char *name,PetscObject obj);
int PetscObjectListDestroy(PetscObjectList *fl);
int PetscObjectListFind(PetscObjectList fl,const char *name,PetscObject *obj)
int PetscObjectListDuplicate(PetscObjectList fl,PetscObjectList *nl);

The function PetscObjectListAdd() will create the initial PetscObjectList if the argument fl points to a NULL.

The PETSc object compose() and query() functions are as follows (defined in src/sys/objects/inherit.c).

PetscErrorCode PetscObjectCompose_Petsc(PetscObject obj,const char *name,PetscObject ptr)
{
  PetscFunctionBegin;
  PetscCall(PetscObjectListAdd(&obj->olist,name,ptr));
  PetscFunctionReturn(PETSC_SUCCESS);
}

PetscErrorCode PetscObjectQuery_Petsc(PetscObject obj,const char *name,PetscObject *ptr)
{
  PetscFunctionBegin;
  PetscCall(PetscObjectListFind(obj->olist,name,ptr));
  PetscFunctionReturn(PETSC_SUCCESS);
}

Compose and Query Functions

PETSc allows you to compose functions by specifying a name and function pointer. In src/sys/dll/reg.c, PETSc defines the following linked list structure.

struct _n_PetscFunctionList {
  void              (*routine)(void);    /* the routine */
  char              *name;               /* string to identify routine */
  PetscFunctionList next;                /* next pointer */
  PetscFunctionList next_list;           /* used to maintain list of all lists for freeing */
};

Each PETSc object contains a PetscFunctionList object. The composefunction() and queryfunction() are given by the following.

PetscErrorCode PetscObjectComposeFunction_Petsc(PetscObject obj,const char *name,void *ptr)
{
  PetscFunctionBegin;
  PetscCall(PetscFunctionListAdd(&obj->qlist,name,fname,ptr));
  PetscFunctionReturn(PETSC_SUCCESS);
}

PetscErrorCode PetscObjectQueryFunction_Petsc(PetscObject obj,const char *name,void (**ptr)(void))
{
  PetscFunctionBegin;
  PetscCall(PetscFunctionListFind(obj->qlist,name,ptr));
  PetscFunctionReturn(PETSC_SUCCESS);
}

In addition to using the PetscFunctionList mechanism to compose functions into PETSc objects, it is also used to allow registration of new class implementations; for example, new preconditioners.

PETSc code that calls composed functions should be done via

PetscUseMethod(obj,"method",(Argument types),(argument variables));
PetscTryMethod(obj,"method",(Argument types),(argument variables));

For example,

PetscErrorCode  KSPGMRESSetRestart(KSP ksp, PetscInt restart)
{
  PetscFunctionBegin;
  PetscValidLogicalCollectiveInt(ksp,restart,2);

  PetscTryMethod(ksp,"KSPGMRESSetRestart_C",(KSP,PetscInt),(ksp,restart));
  PetscFunctionReturn(PETSC_SUCCESS);
}

The Try variant skips the function call if the method has not been composed with the object while the Use version generates an error in that case. See also, PetscUseTypeMethod(), and PetscTryTypeMethod().

Simple PETSc Objects

Some simple PETSc objects do not need PETSCHEADER and the associated functionality. These objects are internally named as _n_<class> as opposed to _p_<class>, for example, _n_PetscFunctionList vs _p_Vec.

PETSc Packages

The PETSc source code is divided into the following library-level packages: sys, Vec, Mat, DM, KSP, SNES, TS, TAO. Each of these has a directory under the src directory in the PETSc tree and, optionally, can be compiled into separate libraries. Each package defines one or more classes; for example, the KSP package defines the KSP and PC classes, as well as several utility classes. In addition, each library-level package may contain several class-level packages associated with individual classes in the library-level package. In general, most “important” classes in PETSc have their own class level package. Each package provides a registration function XXXInitializePackage(), for example KSPInitializePackage(), which registers all the classes and events for that package. Each package also registers a finalization routine, XXXFinalizePackage(), that releases all the resources used in registering the package, using PetscRegisterFinalize(). The registration for each package is performed “on demand” the first time a class in the package is utilized. This is handled, for example, with code such as

PetscErrorCode  VecCreate(MPI_Comm comm, Vec *vec)
{
  Vec            v;

  PetscFunctionBegin;
  PetscValidPointer(vec,2);
  *vec = NULL;
  VecInitializePackage();
  ...