PETSc
USER MEETING 2016

TUESDAY, JUNE 28

08:00 - 09:00 – Registration
09:00 - 09:30 – Welcome
09:30 - 10:30 – PETSc Tutorial (About, Design Philosophy, Installation)
10:30 - 11:00 – Coffee Break
11:00 - 12:30 – PETSc Tutorial (Regular Meshes, Linear Solvers, Preconditioners)
12:30 - 14:00 – Lunch Break
14:00 - 15:30 – PETSc Tutorial (Nonlinear Solvers, Timesteppers)
15:30 - 16:00 – Coffee Break
16:00 - 17:30 – PETSc Tutorial (Unstructured Meshes, Threads and Accelerators)

Slides: [Part 1] [Part 2]

WEDNESDAY, JUNE 29

08:30 - 09:00 – Registration
09:00 - 09:30 – Algebraic Multigrid Preconditioners: When and When Not to Use Them (Mark Adams, Lawrence Berkeley National Laboratory) [Abstract] [Slides]
09:30 - 10:00 – Extreme-Scale Multigrid Components within PETSc (Patrick Sanan, USI Lugano) [Abstract] [Slides]
10:00 - 10:30 – A Massively Parallel Multigrid Solver using PETSc and Unstructured Meshes on Tier0 Supercomputers (Hugues Digonnet, Institut de Calcul Intensif - École Centrale de Nantes) [Abstract] [Slides]
10:30 - 11:00 – Coffee Break
11:00 - 11:30 – Early Experiences and Future Directions with the 2nd Generation ("Knights Landing") Xeon Phi Processor and PETSc (Richard Mills, Intel) [Abstract] [Slides]
11:30 - 12:00 – Overview of SLEPc and Recent Additions (Jose Roman, Universitat Politècnica de València) [Abstract] [Slides]
12:00 - 12:30 – PERMON Libraries for Large Scale Quadratic Programming (Vaclav Hapla, Technical University of Ostrava) [Abstract] [Slides]
12:30 - 14:00 – Lunch Break
14:00 - 14:30 – Accelerating the Shifted Laplacian for the Helmholtz Equation using Deflation and Multigrid Vectors (Domenico Lahaye, TU Delft) [Abstract] [Slides]
14:30 - 15:30 – Plenary Discussion: To thread or not to thread? Why PETSc favors MPI-only [Slides]
15:30 - 16:00 – Coffee Break
16:00 - 17:30 – Unconference: User-Driven Focus Groups

THURSDAY, JUNE 30

09:00 - 09:30 – Oceanic CO2 Uptake - Use of PETSc in Climate Research (Jaroslaw Piwonski, Christian-Albrechts-Universität zu Kiel) [Abstract] [Slides]
09:30 - 10:00 – Numerical Simulations of CO2 Geo-Sequestration using PETSc (Henrik Buesing, RWTH Aachen University) [Abstract] [Slides]
10:00 - 10:30 – Big Simulation in Tunnel Engineering Application (Hoang Guang Bui, Ruhr University Bochum) [Abstract] [Slides]
10:30 - 11:00 – Coffee Break
11:00 - 11:30 – Salvus: A Flexible Open-Source Package for Full-Waveform Modelling and Inversion (Michael Afanasiev, ETH Zürich) [Abstract] [Slides]
11:30 - 12:00 – Modeling of Interface Decohesion in Polycrystalline Microstructures (Luv Sharma, TU Eindhoven) [Abstract] [Slides]
12:00 - 12:30 – Bringing PETSc to the Multi-Scale Simulation of Newtonian and non-Newtonian Free-Surface Flows (Juan Luis Prieto, Universidad Politécnica de Madrid) [Abstract] [Slides]
12:30 - 14:00 – Lunch Break
14:00 - 15:30 – Panel: How can PETSc Lower the Entry Barrier?
15:30 - 16:00 – Coffee Break
16:00 - 17:30 – PETSc User Meeting Feedback Session: What to Learn from this Meeting and Planning the Future

POSTERS

• Fluid Interface Detection with PETSc and DONLP2 (Alin-Adrian Anton, Polytechnic University of Timisoara) [Abstract] [Poster] [Slides]
• PERMON libraries for Large Scale Quadratic Programming (Vaclav Hapla, IT4Innovations National Supercomputing Center) [Abstract] [Poster]
• PETSc Application in Computation of Parachute Deceleration System (Xiaolin Li, Stony Brook University) [Abstract] [Poster]
• Classical Density Functional Theory for Liquid-Fluid Interfaces using modified iSAFT and the heterosegmented GC-PC-SAFT Equation of State (Jonas Mairhofer, University of Stuttgart) [Abstract] [Poster]
• Software Library for Scalable Multi-Physics Multi-Scale Network Simulation: Application to Water Distribution Systems (Daniel A. Maldonado, Illinois Institute of Technology) [Abstract] [Poster]
• A Parallel Modeling Tool for Lithospheric Deformation (Anton Popov, University of Mainz) [Abstract] [Poster]
• Causality Inference in a Nonstationary and Nonhomogenous Framework (Lukas Pospisil, USI Lugano) [Abstract] [Poster]
• Physically Preconditioned, HPC-Enabled Fluid-Structure Interaction Modeling (Frederico Teixeira, IT'IS Foundation & D-ITET / ETHZ) [Abstract] [Poster]
• petsc-hs: Haskell Bindings for PETSc (Marco Zocca, TU Delft) [Abstract] [Poster]

ABSTRACTS

Algebraic Multigrid Preconditioners: When and When Not to Use Them (Mark Adams, Lawrence Berkeley National Laboratory) [Slides]
I will provide some background of algebraic multigrid (AMG) methods with the intent of providing algorithmic foundation to discuss using AMG solvers in general and the native PETSc AMG solver (-pc_type gamg). I will discuss operators and discretization issues that affect AMG solver performance and go over some of the common performance debugging taxonomy process that is involved in optimizing standard AMG code parameters. (I will not discuss special purpose AMG methods.)

Extreme-Scale Multigrid Components within PETSc (Patrick Sanan, USI Lugano) [Slides]
Elliptic partial differential equations (PDEs) frequently arise in continuum descriptions of physical processes relevant to science and engineering. Multilevel preconditioners represent a family of scalable techniques for solving discrete PDEs of this type and thus are the method of choice for highresolution simulations. The scalability and time-to-solution of massively parallel multilevel preconditioners can be adversely affected by using a coarse-level solver with sub-optimal algorithmic complexity. To maintain scalability, agglomeration techniques applied to the coarse level have been shown to be necessary. In this work, we present a new software component introduced within the Portable Extensible Toolkit for Scientific computation (PETSc) which permits agglomeration. We provide an overview of the design and implementation of this functionality, together with several use cases highlighting the benefits of agglomeration. Lastly, we demonstrate via numerical experiments employing geometric multigrid with structured meshes, the flexibility and performance gains possible using our MPI-rank agglomeration implementation.

A Massively Parallel Multigrid Solver using PETSc and Unstructured Meshes on Tier0 Supercomputers (Hugues Digonnet, Institut de Calcul Intensif - École Centrale de Nantes) [Slides]
Today's supercomputers have hundred of thousand cores, hundred of TB of memory and allow us to solve very large systems. To solve such systems a multigrid method must be used to keep the algorithmic complexity close to linear. Here we present a multigrid method implementation based on unstructured meshes using the framework provided by PETSc. Weak scalability results will be presented up to 262 144 cores on JuQUEEN and 65 536 cores on Curie, leading to the solution of a global linear system for the Stokes equation of 100 billion of unknowns using 200 TB of memory.

Early Experiences and Future Directions with the 2nd Generation ("Knights Landing") Xeon Phi Processor and PETSc (Richard Mills, Intel) [Slides]
As the high-performance computing community pushes towards the exascale horizon, power and heat considerations have driven the increasing importance and prevalence of fine-grained parallelism in new computer architectures. The Intel Many Integrated Core (MIC) architecture achieves power efficient, high performance by utilizing many small, low power x86 cores within a single Intel Xeon Phi processor. The second generation ("Knights Landing") Intel Xeon Phi Processor offers a very high degree of parallelism in a single, bootable CPU, with up to 72 cores (288 hardware threads), each of which has two vector processing units supporting 512-bit SIMD operations. It also incorporates very high-bandwidth on-package memory, which facilitates good utilization of those cores when working with the assembled sparse matrices typical of many PETSc-based applications. We will discuss the key architectural features of this new processor, relate early experiences running PETSc on it, and discuss possible ways that PETSc should evolve to best utilize this and future manycore architectures.

Overview of SLEPc and Recent Additions (Jose Roman, Universitat Politècnica de València) [Slides]
SLEPc provides a collection of solvers for various types of eigenvalue problems. Its use in scientific computing projects is steadily growing. We review the functionality currently offered in SLEPc, from basic to more advanced solution in the context of linear eigenproblems as well as nonlinear and other kinds of problems.

PERMON Libraries for Large Scale Quadratic Programming (Vaclav Hapla, Technical University of Ostrava) [Slides] [Poster]
PERMON forms a collection of software libraries, uniquely combining quadratic programming (QP) algorithms and domain decomposition methods (DDM), built on top of the well-known PETSc framework for numerical computations. Among the main applications are contact problems of mechanics. Our PermonFLLOP package is focused on non-overlapping DDM of the FETI type, allowing efficient and robust utilization of contemporary parallel computers for problems with billions of unknowns. Any FEM software can be used to generate mesh and assemble the stiffness matrices and load vectors per each subdomain independently. Additionally, a mapping from the local to the global numbering of degrees of freedom is needed, and non-penetration and friction information in case of contact problems. All these data are passed to PermonFLLOP, which prepares auxiliary data needed in the DDM. PermonQP, a general purpose QP solver, is then called in the backend to solve the resulting equality constrained problem with additional inequality constraints in case of contact problems. PermonQP can be used also standalone for applications where QP arise like least-squares regression, data fitting, data mining, support vector machines, control systems, and others. It can potentially wrap TAO solvers and, the other way around, extend TAO with possibility to solve problems with linear equality and/or inequality constraints.

Accelerating the Shifted Laplacian for the Helmholtz Equation using Deflation and Multigrid Vectors (Domenico Lahaye, TU Delft) [Slides]
Many important physical phenomena can be described by the Helmholtz equation. We investigate to what extent the convergence of the shifted Laplacian preconditioner for the Helmholtz equation can be accelerated using deflation and multigrid vectors. We therefore present a unified framework for two published algorithmic variants. The first deflates the preconditioned operator while the second combines the deflation and the preconditioner in a multiplicative fashion. A Rigorous Fourier Analysis shows that in both variants the deflation operator clusters the spectrum of the preconditioned operator. Our analysis also reveals that the near-singularity of the coarse grid operator causes the spread of the eigenvalues to increase with the frequency. This effect is less pronounced in the first algorithmic variant at the expense of a higher computational cost. Numerical experiments using the second algorithmic variant on three-dimensional test cases show that deflation reduces the iteration count considerably and results in a speedup of a factor between six and ten.

Oceanic CO2 Uptake - Use of PETSc in Climate Research (Jaroslaw Piwonski, Christian-Albrechts-Universität zu Kiel) [Slides]
We present the Marine Ecosystem Toolkit for Optimization and Simulation in 3D. It is based on a parallel, high-performance software for the investigation of periodic steady states of biogeochemical models driven by a global ocean circulation. Here, we use parallel PETSc data types and parallel matrix-vector multiplication as well as the PETSc Newton-Krylov solver. Moreover, the simulation has been ported to GPUs using a commercial Portland Fortran GPU compiler. Additionally, we use Automatic Differentiation techniques for an adjoint version of the model for simultaneous optimization of state and parameter variables.

Numerical Simulations of CO2 Geo-Sequestration using PETSc (Henrik Buesing, RWTH Aachen University) [Slides]
The geological sequestration of CO2 is considered as one option to mitigate anthropogenic effects of CO2 emissions on climate. We consider a system of non-linear, strongly coupled, partial differential equations describing multiphase CO2-brine fluid flow in the subsurface. Dissolution processes of CO2 in brine and water in CO2 are taken into account. After discretization in space and time, we linearize the arising non-linear algebraic systems with Newton’s method using PETSc SNES and solve the resulting linear systems with different direct and iterative linear solvers using PETSc KSP. We will show simulation results of CO2 injection into a highly heterogeneous, permeable sandstone including dissolution effects.

Big Simulation in Tunnel Engineering Application (Hoang Guang Bui, Ruhr University Bochum) [Slides]
In this talk, a High Performance Computing framework for tunnel simulation is presented. The framework uses classical Finite Element method coupled with state of the art parallel software library (Metis, PETSc) to deliver fast solution. The data from the simulation is then used for development of a substitute model, which provides real-time prediction capability of ground settlement. In the numerical examples, selected simulations from industry tunnel project (Werhahn Linie von Düsseldorf) are presented.

Salvus: A Flexible Open-Source Package for Full-Waveform Modelling and Inversion (Michael Afanasiev, ETH Zürich) [Slides]
Within all domain-specific software projects, finding the correct balance between flexibility and performance is often difficult. In the seismic imaging community, the trend has been to move towards codes which are heavily optimized, but which often sacrifice usability and flexibility. Here we introduce Salvus: an open-source HPC high-order finite element (FE) package focused on full-waveform modelling and inversion, which is designed to be both flexible and performant.
Salvus was constructed by following modern software design practices, testing protocols, and by establishing its foundations upon existing open-source high-level scientific libraries, such as PETSc and Eigen. The FE framework is generalized over spatial dimensions, time-integrators, polynomial order and wave-propagation physics, and provides support for both hexahedral and tetrahedral meshes. Additionally, support is included for various numerical optimization methods. We discuss our usage of existing open-source scientific libraries, our choice level of abstraction, and quantitatively investigate the performance penalties associated with these abstractions.

Modeling of Interface Decohesion in Polycrystalline Microstructures (Luv Sharma, TU Eindhoven) [Slides]
In this work, we propose and illustrate the implementation of interface decohesion using the concepts of FFT based field solvers and eigen strain based damage mechanics. The FFT based field solver is utilized to solve the partial differential equations of static mechanical equilibrium and that of damage regularization. A strong coupling between the two field equations is attained by a staggered approach. The anisotropic kinematics of interface separation and sliding is accounted for by introduction of second order damage deformation gradient in the partition of the total deformation gradient. The deformation rate in the opening and sliding modes is dependent on the geometric normals to the interfaces which are calculated using first order cartesian moments. This work is implemented in the open source code DAMASK, where in, the governing partial differential equations are solved using PETSc.

Bringing PETSc to the Multi-Scale Simulation of Newtonian and non-Newtonian Free-Surface Flows (Juan Luis Prieto, Universidad Politécnica de Madrid) [Slides]
In this presentation, we describe our micro-macro, multi-scale approach to non-Newtonian free-surface flows. The method uses, at the macroscopic scale, a particle level-set technique along with a semi-Lagrangian, finite element discretization; the microscopic scale is described via variance-reduced, Brownian Dynamics simulations of Hook (Oldroyd-B) and FENE dumbbells. The resulting Stokes problems including the (polymer) extra-stress tensor are efficiently solved using PETSc. We show numerical tests in the rising bubble configuration, for Newtonian and non-Newtonian (Hooke, FENE) fluids, which run in a commodity, multi-core PC.

Fluid Interface Detection with PETSc and DONLP2 (Alin-Adrian Anton, Polytechnic University of Timisoara) [Poster] [Slides]
Fluid engineering problems require tailored numerical computation algorithms. The DONLP2 library is used for the SQP optimization of a function which tracks the fluid interface using a finite element method implemented in PETSc. Several other software packages are combined in order to build the computational domain and to generate the unstructured mesh. The challenges for developing an automated tool are discussed. In particular, the fluid interface between stagnant regions and the main swirling flow is detected using the optimization algorithm.

PETSc Application in Computation of Parachute Deceleration System (Xiaolin Li, Stony Brook University) [Poster]
In this project, we use the front tracking method on a dual-stiffness spring-mass system to model the dynamic evolution of parachute canopy and risers. Our spring-mass model includes both tensile-stiffness and angular-stiffness, which has the capability of duplicating the realistic strain and stress of the elastic membrane and conforms with both Young's modulus and Poisson ratio. This mechanical structure is coupled with the incompressible turbulence fluid solver through the "impulse method". In addition, GPU-based parallel computing techniques are applied to accelerate the computational speed and increase the resolution of numerical results. PETSc is used as the linear solver in solving the incompressible Navier-Stokes equation through projection method.

Classical Density Functional Theory for Liquid-Fluid Interfaces using modified iSAFT and the heterosegmented GC-PC-SAFT Equation of State (Jonas Mairhofer, University of Stuttgart) [Poster]
Liquid-fluid interfaces play a determining role in many technical applications. The properties of these inhomogeneous systems are strongly influenced by surfactants. Surface-active substances consist of distinct functional groups with opposing chemical properties. In order to study the effect of such substances on interfacial properties, detailed knowledge of the distribution and orientation of these molecules at the interface is necessary.
We have implemented a classical DFT formalism which enables us to determine the density profiles of single functional groups of real components across liquid-fluid interfaces and, therefore, study the local assembly of molecules like surfactants at the interface. The DFT approach predicts interfacial tensions of pure components and multicomponent mixtures in good agreement with experimental values.

Software Library for Scalable Multi-Physics Multi-Scale Network Simulation: Application to Water Distribution Systems (Daniel A. Maldonado, Illinois Institute of Technology) [Poster]
We present an application of DMNetwork , a software library for scalable network simulation included in the scientific computing library PETSc, on transient analysis of water distribution system. We use DMNetwork for the development of WASH, a scalable simulator for water distribution systems that includes high-fidelity physical model couplings.

A Parallel Modeling Tool for Lithospheric Deformation (Anton Popov, University of Mainz) [Poster]
We describe a software framework LaMEM (Lithosphere and Mantle Evolution Model) focused on the thermo-mechanically coupled modeling of 3D lithospheric deformation. The code employs a staggered grid finite difference discretization together with a marker in cell approach. To achieve scalability on massively parallel computers, we use the discretization objects and solvers from PETSc library. The coupled nonlinear equations are solved by the Jacobian Free Newton Krylov method preconditioned with Galerkin geometric multigrid. We also derive the analytical Jacobian expressions for the visco-elasto-plastic rheology. The code is tested by a range of benchmarks and example problems.

Causality Inference in a Nonstationary and Nonhomogenous Framework (Lukas Pospisil, USI Lugano) [Poster]
The project deploys statistical and computational techniques to develop a novel approach to causality inference in multivariate time series of economical data on equity and credit risks. Emerging causality framework is implemented through a clustering of time series based on a minimization of the averaged clustering functional, which describes the mean distance between observation data and its representation in terms of given number of abstract Bayesian causality models of a certain predefined class. One of the most challenging components is a Quadratic Programming problem with the special structure. We demonstrate and compare solving practical benchmark problems.

Physically Preconditioned, HPC-Enabled Fluid-Structure Interaction Modeling (Frederico Teixeira, IT'IS Foundation & D-ITET / ETHZ) [Poster]
The AneuX project investigates aneurysm evolution, rupture risk, and remodeling in a combined experimental-numerical approach. A semi-implicit method was developed to avoid the complexity of monolithic and interface-degradation issues of traditional semi-implicit methods. A physically motivated two-level Schur-complement preconditioning approach was developed to facilitate modeling of realistic problems. For increased boundary condition realism, the 3D FSI interfaces with a 1D compliant vascular network model.
The solver integrates with the Sim4Life platform, benefitting from a generic PETSc-based, mixed element FEM-solving framework. It incorporates a novel image-based vasculature modeling technique with tunable interactivity, as well as modeling, scripting, analysis, and visualization functionality.

petsc-hs: Haskell Bindings for PETSc (Marco Zocca, TU Delft) [Poster]
Scientific programming is the craft of representing a physical phenomenon, or an algorithm that interacts with it, in form that can be executed on a computer. Correctness and performance arise as the primary concerns, however good scientific practice demands an experiment to be easily accessible to peer review. One approach to solving this conundrum is to embed high-performance numerical codes into a high-level language that lets users abstract away the necessary book-keeping. PETSc-hs is an ongoing effort to embed PETSc in the purely-functional, statically-typed language Haskell; the type system assists the user in ensuring program correctness before its execution, and enables algebraic composition of functionality.

[Back to PETSc User Meeting 2016 main page]