1 Rendering and Simulation Libraries Used
The rendering system used in our framework is developed using scenery [gunther2019scenery], an open-source VR library for both volumetric and geometric data. In simulations, volumetric data typically stem from discretized continuous fields and geometric data from surface meshes or postprocessing results, such as isosurfaces or streamlines. scenery supports rendering to both commodity VR headsets and distributed setups, such as VR CAVEs and PowerWalls. It uses the high-performance Vulkan API for rendering, and therefore has the potential to harness the power of modern GPUs better than traditional OpenGL-based solutions, as still predominantly used by in situ solutions like ParaView Catalyst [ayachit2015paraview] and VisIt Libsim [kuhlen2011parallel].
The simulation engine used in our framework is OpenFPM [incardona2019openfpm], an open-source library that enables rapid development of scalable and efficient particle- and mesh-based numerical simulations. OpenFPM largely automates parallelization on high-performance computers, including distributed-memory systems, many-core systems, and GPGPUs. It has been used to implement simulations ranging from Smoothed-Particle Hydrodynamics to Molecular Dynamics to Discrete Element Methods on thousands of CPU cores [incardona2019openfpm].
Integrating our work into OpenFPM helps us reach a wide audience of computational scientists who benefit from interactive in situ visualization and steering, but may lack the time or expertise to write scalable solutions from scratch. We implement in situ functionality into OpenFPM in a manner that allows users to enable it in a fraction of the time and effort it normally takes to instrument a simulation with an in situ solution.
2 In Situ Architecture
We design our in situ architecture targeting a rendering frame rate of at least 60 Hz, rendering latency on viewpoint changes of at most 20 ms, and low interaction latency for steering commands.
Key to achieving these goals is to minimize data movement. We thus choose a tightly-coupled structure with simulation and rendering sharing resources, running on the same nodes within a cluster. Figure 1 shows a schematic. Each compute node runs the distributed OpenFPM environment for the simulation, along with a scenery-based application that performs local rendering. All simulation data generated on a node are rendered on the node itself. The simulation data are shared between OpenFPM and scenery via a common pointer to shared memory. This enables zero-copy rendering.
Several in situ solutions, including ParaView Catalyst [ayachit2015paraview] and VisIt Libsim [kuhlen2011parallel], adopt a synchronous execution strategy, where simulation and visualization run sequentially. We choose an asynchronous strategy, aiming to reduce interaction and rendering latency. In our asynchronous strategy, neither simulation nor visualization need to wait for the other, which means that steering commands can be incorporated at the earliest, and rendering can begin while the simulation is in progress. Visualization artifacts that may result from this lack of synchronization are expected to be small, since numerical stability requires the simulation to only change slightly between time steps. For our application, we emphasize low latency over these minor visual artifacts.
The rendering results produced on the compute nodes are then composited using a divide-and-conquer strategy similar to the binary swap algorithm [ma1994parallel]. The scenery process on the Head Node is responsible for streaming the composited result to a remote client, as well as for receiving requests for changes in visualization parameters. The remote client also communicates steering commands to the OpenFPM process on the Head Node, which conveys them to all OpenFPM processes via MPI messages.
In contrast to other approaches, we intend to render an explorable representation, such as a Volumetric Depth Image (VDI) [frey2013explorable], instead of a simple image. Such representations of volumetric data can, once generated, be reprojected to a new viewpoint without redoing the ray casting. We intend to render such a representation on the cluster and stream it to the remote client. Small viewpoint changes can then be handled locally by the remote client without involving network latency. This is crucial for meeting the latency requirements of VR, where viewpoint changes occur frequently.
3 Preliminary Results
We have implemented a prototype of our framework, performing particle-as-sphere rendering of a distributed OpenFPM simulation. The simulation runs parallel on multiple CPUs, while visualization is done on the GPU. The shared-memory communication between the processes on a single node is extended by a protocol to handle dynamic memory requests and reallocations. The scenery application reads the simulation data from shared memory and understands the data structures used by OpenFPM. Our implementation is open source. The scenery application can be found at whitegithub.com/scenerygraphics/scenery-insitu and the OpenFPM project at whitegit.mpi-cbg.de/openfpm/openfpm_pdata (in the “insitu_visualization” branch).
Figure 2 shows our prototype applied to a Molecular Dynamics simulation. The simulation code written using OpenFPM is only 161 lines of C++, and the inclusion of in situ visualization requires changes in only two lines.
4 Future Work and Conclusions
We have presented a framework architecture for interactive VR in situ visualization of parallel numerical simulations. We used scenery and OpenFPM to base the rendering and simulation parts of the framework on. We have implemented a first prototype of the system, demonstrating the feasibility of achieving our design goals. In our prototype, enabling live in situ visualization only required minimal changes in the existing simulation code.
Our prototype does not yet meet the targets we set for immersive in situ visualization in Section 2, but it enables us to identify a number of challenges that should be addressed in order to meet them: First, volumetric rendering results need to be in an explorable representation, such as a VDI, as explained in Section 2. Second, we need a solution to efficiently generate and composit such a representation on a cluster. Third, support needs to be added for simulations running on GPGPUs. In addition, we are going to extend our framework to NUI-based steering by instrumenting OpenFPM to receive steering commands on the fly and by expanding scenery’s support for NUI interaction.
Taken together, we are confident that many applications in scientific computing will benefit from a more immersive and more explorative mode of interaction.