Parallel Dataflow Coupling
Computational science involves a workflow of interconnected tasks, several of
them being executed on a supercomputer, for example simulation, analysis,
visualization, or user interaction. A workflow can be modeled as a directed
graph, where the nodes of the graph are tasks, and the edges represent
information exchanged between the tasks. We assume that the graph can have
cycles; feedback loops can exist between tasks so that the result of a
downstream task can be used to modify an upstream one (sometimes called
computational steering).
We define dataflow as the parallel information exchange between tasks in a workflow.
Traditionally, tasks exchange data through files. However, the growing mismatch
between HPC computing rate and I/O bandwidth motivates shifting from
file-oriented workflow models to in situ workflows, where dataflows are through
memory or the supercomputer interconnect, avoiding the storage I/O bottleneck.
We designed our dataflow as a middleware layer below the workflow engine
so that one dataflow library can support multiple workflow tools simultaneously.
However, several challenges need to be solved as a result of this design choice.
We support two different in situ coupling modes: time division and space division. In time
division, two tasks, producer and consumer, share the same resources and run
sequentially in time. In space division, the producer and consumer run on different
resources concurrently in time. A workflow can mix modes for each
pair of coupled tasks.
In space division mode, disparate data rates between producer and consumer tasks may
require buffering in order for both tasks to proceed as quickly as possible.
In the following use case, molecular dynamics simulation is coupled with density
estimation and isosurfacing in a 3-step in situ pipeline. Strong scaling is
improved considerably using Decaf compared with a previous workflow tool.
The following use case is a complex workflow involving molecular dynamics,
interactive computational steering, and 2 visualization tools. The overhead of
Decaf's put/get communication is minimal.
The goal of the following workflow is the in situ conversion of particles from an
N-body cosmology simulation to particle density over a regular
2D and 3D grid, using a Voronoi tessellation as an intermediate step.
The dataflow programming model is implemented in our Decaf software.
