Changes: 3.16

General:

• Change MPIU_Allreduce() to always returns a MPI error code that should be checked with CHKERRMPI(ierr)

• Add support for A64FX and Cray compilers

• Add support for ESSL 5.2 and later; drop support for ESSL <=5.1

• Add support for NEC-SX Tsubasa Vector Engine

• Add support for NVIDIA HPC SDK

Configure/Build:

• Configure requires Python-2.7 or Python-3.4+.

• Remove --with-kokkos-cuda-arch. One can use -with-cuda-gencodearch to specify the cuda arch for Kokkos. Usually not needed since PETSc auto detects that

• For --download-hdf5, disable --download-hdf5-fortran-bindings by default

• Add OpenCascade package to PETSc and allow --download-opencascade

• Add support for hypre in device mode for both NVIDIA and AMD GPUs

• Extend detection of C++ dialect to C++17. Configure now also takes minimum and maximum required C++ dialect of packages into account when choosing the C++ dialect

Sys:

• Add PetscDevice class to manage discovered GPU devices

• Add PetscDeviceKind

• Add PetscDeviceCreate(), PetscDeviceConfigure(), and PetscDeviceDestroy()

• Add PetscDeviceContext class to manage asynchronous GPU compute support via a fork-join model

• Add PetscStreamType and PetscDeviceContextJoinMode

• Add PetscDeviceContextCreate(), PetscDeviceContextDestroy(), PetscDeviceContextSetDevice(), PetscDeviceContextGetDevice(), PetscDeviceContextSetStreamType(), PetscDeviceContextGetStreamType(), PetscDeviceContextSetUp(), PetscDeviceContextDuplicate(), PetscDeviceContextQueryIdle(), PetscDeviceContextWaitForContext(), PetscDeviceContextFork(), PetscDeviceContextJoin(), PetscDeviceContextSynchronize(), PetscDeviceContextGetCurrentContext(), PetscDeviceContextSetCurrentContext(), and PetscDeviceContextSetFromOptions()

• Deprecate petsccublas.h and petschipblas.h in favor of petscdevice.h and petscdevicetypes.h

• Add GPU event timers to capture kernel execution time accurately

• Remove WaitForCUDA() and WaitForHIP() before PetscLogGpuTimeEnd()

• Add MPIU_REAL_INT and MPIU_SCALAR_INT datatypes to be used for reduction operations

• Add MPIU_MAXLOC and MPIU_MINLOC operations

• Add CHKERRCXX() to catch C++ exceptions and return a PETSc error code

• Remove PetscStack routines from public headers, this class should now be considered private

PetscViewer:

• PetscViewerHDF5PushGroup(): if input path begins with /, it is taken as absolute, otherwise relative to the current group

• Add PetscViewerHDF5HasDataset()

• PetscViewerHDF5HasAttribute(), PetscViewerHDF5ReadAttribute(), PetscViewerHDF5WriteAttribute(), PetscViewerHDF5HasDataset() and PetscViewerHDF5HasGroup() support absolute paths (starting with /) and paths relative to the current pushed group

• Add input argument to PetscViewerHDF5ReadAttribute() for default value that is used if attribute is not found in the HDF5 file

• Add PetscViewerHDF5PushTimestepping(), PetscViewerHDF5PopTimestepping() and PetscViewerHDF5IsTimestepping() to control timestepping mode.

• One can call PetscViewerHDF5IncrementTimestep(), PetscViewerHDF5SetTimestep() or PetscViewerHDF5GetTimestep() only if timestepping mode is active

• Error if timestepped dataset is read/written out of timestepping mode, or vice-versa

PetscDraw:

AO:

IS:

VecScatter / PetscSF:

PF:

Vec:

• Add VecMean() to calculate arithmetic mean of elements of a vector

• Add VecBoundToCPU() to query information set with VecBindToCPU()

PetscSection:

• Extend PetscSectionView() for section saving to HDF5

• Add PetscSectionLoad() for section loading from HDF5

PetscPartitioner:

Mat:

• MATMPIKAIJ now tracks the object state of the AIJ matrix describing the blockwise action of the KAIJ matrix and automatically rebuilds internal data structures before executing operations with the KAIJ matrix if the state has changed

• Factorization types now provide their preferred ordering (which may be MATORDERINGEXTERNAL) to prevent PETSc PCFactor from, by default, picking an ordering when it is not ideal

• Deprecate MatFactorGetUseOrdering(); Use MatFactorGetCanUseOrdering() instead

• Add --download-htool to use hierarchical matrices with the new type MATHTOOL

• Add MATCENTERING special matrix type that implements action of the centering matrix

• Remove -mat_mumps_icntl_7 1 option, use -pc_factor_mat_ordering_type <type> to have PETSc perform the ordering (sequential only)

• Add MATSOLVERSPQR - interface to SuiteSparse QR factorization

• Add MatSeqAIJKron() - Kronecker product of two MatSeqAIJ

• Add MatNormalGetMat() to retrieve the underlying Mat of a MATNORMAL

• Add MatNormalHermitianGetMat() to retrieve the underlying Mat of a MATNORMALHERMITIAN

• Add VecCreateMPICUDA() and VecCreateMPIHIP() to create MPI device vectors

• Add accessor routines for device index data of MATSEQAIJCUSPARSE matrices: MatSeqAIJCUSPARSEGetIJ() and MatSeqAIJCUSPARSERestoreIJ()

• Add accessor routines for device data of MATSEQAIJCUSPARSE matrices: MatSeqAIJCUSPARSEGetArray(), MatSeqAIJCUSPARSERestoreArray(), MatSeqAIJCUSPARSEGetArrayRead(), MatSeqAIJCUSPARSERestoreArrayRead(). MatSeqAIJCUSPARSEGetArrayWrite(), MatSeqAIJCUSPARSERestoreArrayWrite()

• Add support for MATHYPRE matrices on NVIDIA and AMD GPUs

• MatPreallocatorPreallocate performance significantly improved

• Add MatGetColumnReductions() developer routine to calculate reductions over columns of a matrix

• Add MatGetColumnSums(), MatGetColumnSumsRealPart(), MatGetColumnSumsImaginaryPart() to compute sums over matrix columns

• Add MatGetColumnMeans(), MatGetColumnMeansRealPart(), MatGetColumnMeansImaginaryPart() to compute arithmetic means over matrix columns

• Add MatBoundToCPU() to query information set with MatBindToCPU()

• Rename MATHARA in MATH2OPUS, supporting distributed memory operations with hierarchical matrices

PC:

• Add PCSetPreSolve()

• Add PCQR - interface to SuiteSparse QR factorization for MatSeqAIJ, MATNORMAL, and MATNORMALHERMITIAN

• Add support for MATNORMAL in PCASM and PCHPDDM

• Add support for BoomerAMG from PCHYPRE to run on NVIDIA and AMD GPUs

• PCShellGetContext() now takes void* as return argument

• Rename PCHARA in PCH2OPUS, supporting distributed memory operations with hierarchical matrices

KSP:

• KSPGetMonitorContext() now takes void* as return argument

• KSPGetConvergenceContext() now takes void* as return argument

SNES:

• Add SNESSetComputeMFFunction()

• Add support for -snes_mf_operator for use with SNESSetPicard()

• SNESShellGetContext() now takes void* as return argument

SNESLineSearch:

TS:

• Add -ts_type irk - fully implicit Runge-Kutta solvers

• Add TSTrajectory interface to the CAMS library for optimal offline checkpointing for multistage time stepping schemes

• Add option -ts_trajectory_memory_type <revolve | cams | petsc> to switch checkpointing schedule software

• Add option -ts_trajectory_max_units_ram to specify the maximum number of allowed checkpointing units

TAO:

• TaoShellGetContext() now takes void* as return argument

DM/DA:

• Change management of auxiliary data in DM from object composition to DMGetAuxiliaryVec()/DMSetAuxiliaryVec(), DMCopyAuxiliaryVec()

• Remove DMGetNumBoundary() and DMGetBoundary() in favor of DS counterparts

• Remove DMCopyBoundary()

• Change interface for DMAddBoundary(), PetscDSAddBoundary(), PetscDSGetBoundary(), PetscDSUpdateBoundary()

• Add DMDAVecGetArrayDOFWrite() and DMDAVecRestoreArrayDOFWrite()

• DMShellGetContext() now takes void* as return argument

DMSwarm:

• Add DMSwarmGetCellSwarm() and DMSwarmRestoreCellSwarm()

DMPlex:

• Add a PETSCVIEWEREXODUSII viewer type for DMView()/DMLoad() and VecView()/VecLoad(). Note that not all DMPlex can be saved in exodusII format since this file format requires that the numbering of cell sets be compact

• Add PetscViewerExodusIIOpen() convenience function

• Add PetscViewerExodusIISetOrder() to generate “2nd order” elements (i.e. tri6, tet10, hex27) when using DMView with a PETSCVIEWEREXODUSII viewer

• Change DMPlexComputeBdResidualSingle() and DMPlexComputeBdJacobianSingle() to take a form key

• Add DMPlexTopologyLoad(), DMPlexCoordinatesLoad(), and DMPlexLabelsLoad() for incremental loading of a DMPlex object from an HDF5 file

• Add DMPlexTopologyView(), DMPlexCoordinatesView(), and DMPlexLabelsView() for incremental saving of a DMPlex object to an HDF5 file

• Add DMPlexSectionView() saving a PetscSection in association with a DMPlex mesh

• Add DMPlexSectionLoad() loading a PetscSection in association with a DMPlex mesh

• Add DMPlexGlobalVectorView() and DMPlexLocalVectorView() saving global and local vectors in association with a data layout on a DMPlex mesh

• Add DMPlexGlobalVectorLoad() and DMPlexLocalVectorLoad() loading global and local vectors in association with a data layout on a DMPlex mesh

• Add DMPlexIsSimplex() to check the shape of the first cell

• Add DMPlexShape to describe prebuilt mesh domains

• Add DMPlexCreateCoordinateSpace() to make an FE space for the coordinates

• Add the automatic creation of a Plex from options, see DMSetFromOptions()

• The old options for DMPlexCreateBoxMesh() NO LONGER WORK. They have been changed to make the interface more uniform

• Replace DMPlexCreateSquareBoundary() and DMPlexCreateCubeBoundary() with DMPlexCreateBoxSurfaceMesh()

• Remove DMPlexCreateReferenceCellByType()

• The number of refinements is no longer an argument to DMPlexCreateHexCylinderMesh()

• Add DMSetLabel()

• Replace DMPlexComputeJacobianAction() with DMSNESComputeJacobianAction()

• Add DMSNESCreateJacobianMF()

• Change DMPlexComputeBdResidualSingle() to take PetscFormKey instead of explicit label/value/field arguments

• Add DMPlexInflateToGeomModel() which pushes refined points out to a geometric boundary

• Separate EGADS and EGADSLite functionality, add DMPlexCreateEGADSLiteFromFile()

• Remove DMPlexReverseCell() and DMPlexOrientCell() in favor of DMPlexOrientPoint()

• Remove DMPlexCompareOrientations() in favor of DMPolytopeMatchOrientation()

• Add DMPlexGetCompressedClosure() and DMPlexRestoreCompressedClosure()

• Add DMPlexMetricCreate as a helper function for creating a (P1) Riemannian metric.

• Add DMPlexMetricCreateUniform as a helper function for creating a uniform metric.

• Add DMPlexMetricCreateIsotropic as a helper function for creating an isotropic metric.

• Add DMPlexMetricEnforceSPD for enforcing that a metric is symmetric positive-definite.

• Add DMPlexMetricNormalize to apply L-p metric normalization.

• Add DMPlexMetricAverage to average an arbitrary number of metrics.

• Add DMPlexMetricAverage2 to average two metrics.

• Add DMPlexMetricAverage3 to average three metrics.

• Add DMPlexMetricIntersection to intersect an arbitrary number of metrics.

• Add DMPlexMetricIntersection2 to intersect two metrics.

• Add DMPlexMetricIntersection3 to intersect three metrics.

FE/FV:

• Change PetscFEIntegrateBdResidual() and PetscFEIntegrateBdJacobian() to take both PetscWeakForm and form key

• Add PetscFEGeomGetPoint() and PetscFEGeomGetCellPoint to package up geometry handling

DMNetwork:

• Add DMNetworkCreateIS() and DMNetworkCreateLocalIS()

• Remove nv from DMNetworkAddSubnetwork()

DMStag:

• Add DMStagStencilToIndexLocal()

DT:

• Add PetscWeakFormCopy(), PetscWeakFormClear(), PetscWeakFormRewriteKeys() and PetscWeakFormClearIndex()

• Add PetscDSDestroyBoundary() and PetscDSCopyExactSolutions()

• PetscDSGetContext() now takes void* as return argument

• Add PetscWeakFormReplaceLabel() to change labels after mesh modification

Fortran:

• Add support for PetscInitialize(filename,help,ierr), PetscInitialize(ierr) in addition to current PetscInitialize(filename,ierr)