[Paper Review] A Modular and Extensible Software Architecture for Particle Dynamics

Sebastian Eibl, Ulrich Rüde|arXiv (Cornell University)|Jun 26, 2019

Distributed and Parallel Computing Systems16 references4 citations

TL;DR

This paper presents MESA-PD, a modular and extensible particle dynamics framework within the waLBerla HPC ecosystem, using code generation and an accessor abstraction to decouple physics kernels from data structures. The approach enables domain scientists to implement and extend interaction models independently of low-level parallelization details, achieving high performance, scalability, and maintainability through automated, type-safe code generation for memory layout, I/O, and MPI communication.

ABSTRACT

Creating a highly parallel and flexible discrete element software requires an interdisciplinary approach, where expertise from different disciplines is combined. On the one hand domain specialists provide interaction models between particles. On the other hand high-performance computing specialists optimize the code to achieve good performance on different hardware architectures. In particular, the software must be carefully crafted to achieve good scaling on massively parallel supercomputers. Combining all this in a flexible and extensible, widely usable software is a challenging task. In this article we outline the design decisions and concepts of a newly developed particle dynamics code MESA-PD that is implemented as part of the waLBerla multi-physics framework. Extensibility, flexibility, but also performance and scalability are primary design goals for the new software framework. In particular, the new modular architecture is designed such that physical models can be modified and extended by domain scientists without understanding all details of the parallel computing functionality and the underlying distributed data structures that are needed to achieve good performance on current supercomputer architectures. This goal is achieved by combining the high performance simulation framework waLBerla with code generation techniques. All code and the code generator are released as open source under GPLv3 within the publicly available waLBerla framework (www.walberla.net).

Motivation & Objective

• Enable domain scientists to implement and extend particle interaction models without deep knowledge of parallel computing or data structures.
• Achieve high performance and strong scaling on modern supercomputers through a modular, extensible software architecture.
• Decouple physics kernels from data storage and communication logic to improve maintainability and portability.
• Support flexible domain partitioning and dynamic particle data structures tailored to specific simulation needs.
• Facilitate coupling with other simulation modules by sharing the same partitioning and data layout through a unified interface.

Proposed method

• Employ a two-stage code generation pipeline: a high-level Python DSL to specify particle properties and interaction models, followed by Jinja template-based C++ code generation.
• Introduce an accessor interface that abstracts particle data access, allowing kernels to operate on any data structure via a uniform API.
• Automatically generate low-level code for MPI serialization, debug output, VTK export, and database I/O based on user-defined particle properties.
• Design interaction kernels as stateless, template-based functors that operate solely through the accessor interface, ensuring independence from data layout.
• Use the waLBerla framework’s parallel infrastructure to ensure high performance and scalability on distributed-memory systems.
• Ensure extensibility by allowing new particle properties and interaction models to be added via configuration, without modifying core framework logic.

Experimental results

Research questions

• RQ1 How can a particle dynamics simulation framework be architected to allow domain scientists to extend physics models without deep knowledge of parallel computing?
• RQ2 What techniques enable high performance and strong scaling while maintaining modularity and extensibility in particle simulations?
• RQ3 How can data structures be dynamically composed per simulation to avoid memory bloat and improve performance?
• RQ4 Can physics kernels be made fully independent of data layout and communication mechanisms through abstraction?
• RQ5 To what extent does automated code generation reduce development time and error rates in HPC particle simulation frameworks?

Key findings

• The code generation pipeline reduces manual coding by automatically producing consistent, type-safe implementations for memory packing, I/O, and MPI communication.
• The accessor abstraction enables physics kernels to be completely decoupled from data layout, allowing reuse across different data structures and frameworks.
• The framework supports flexible domain partitioning, which enhances coupling efficiency and workload balancing in multi-physics simulations.
• Dynamic particle data structures are generated on-demand based on simulation needs, avoiding unused memory and improving performance.
• The separation of concerns through accessors and code generation allows specialists to work independently—domain scientists on physics, HPC experts on performance—without codebase entanglement.
• The approach ensures correctness and consistency across generated functions, eliminating subtle bugs from manual synchronization of data layout and I/O routines.