[Paper Review] Parthenon -- a performance portable block-structured adaptive mesh refinement framework

Philipp Grete, Joshua C. Dolence+13 more|arXiv (Cornell University)|Feb 24, 2022

Advanced Data Storage Technologies4 citations

TL;DR

Parthenon is a performance-portable, block-structured adaptive mesh refinement (AMR) framework built on Kokkos, designed to enable efficient exascale simulations across diverse architectures. It achieves 1.7×10¹³ zone-cycles/s on 9,216 Frontier nodes with ~92% weak scaling efficiency, leveraging device memory allocation, logical packing of variables and blocks, and one-sided asynchronous MPI to minimize data movement and communication overhead.

ABSTRACT

On the path to exascale the landscape of computer device architectures and corresponding programming models has become much more diverse. While various low-level performance portable programming models are available, support at the application level lacks behind. To address this issue, we present the performance portable block-structured adaptive mesh refinement (AMR) framework Parthenon, derived from the well-tested and widely used Athena++ astrophysical magnetohydrodynamics code, but generalized to serve as the foundation for a variety of downstream multi-physics codes. Parthenon adopts the Kokkos programming model, and provides various levels of abstractions from multi-dimensional variables, to packages defining and separating components, to launching of parallel compute kernels. Parthenon allocates all data in device memory to reduce data movement, supports the logical packing of variables and mesh blocks to reduce kernel launch overhead, and employs one-sided, asynchronous MPI calls to reduce communication overhead in multi-node simulations. Using a hydrodynamics miniapp, we demonstrate weak and strong scaling on various architectures including AMD and NVIDIA GPUs, Intel and AMD x86 CPUs, IBM Power9 CPUs, as well as Fujitsu A64FX CPUs. At the largest scale on Frontier (the first TOP500 exascale machine), the miniapp reaches a total of $1.7 imes10^{13}$ zone-cycles/s on 9,216 nodes (73,728 logical GPUs) at ~92% weak scaling parallel efficiency (starting from a single node). In combination with being an open, collaborative project, this makes Parthenon an ideal framework to target exascale simulations in which the downstream developers can focus on their specific application rather than on the complexity of handling massively-parallel, device-accelerated AMR.

Motivation & Objective

• To address the lack of performance-portable, application-level support for exascale architectures in block-structured AMR frameworks.
• To enable efficient, scalable simulations on heterogeneous systems including CPUs and GPUs across multiple exascale platforms.
• To reduce data movement and communication overhead in multi-node AMR simulations through device memory allocation and optimized kernel launches.
• To provide a modular, extensible framework that allows application developers to focus on physics rather than low-level parallelization and portability concerns.
• To demonstrate strong and weak scaling performance across diverse architectures, including AMD and NVIDIA GPUs, Intel and AMD x86 CPUs, IBM Power9, and Fujitsu A64FX CPUs.

Proposed method

• The framework is built on the Kokkos programming model to enable abstraction across diverse hardware architectures.
• All simulation data is allocated directly in device memory to minimize data movement between host and device.
• Variables and mesh blocks are logically packed into larger structures to reduce kernel launch overhead, especially for small block sizes.
• One-sided, asynchronous MPI calls are used to overlap communication with computation and reduce latency in multi-node environments.
• The framework provides abstractions for multidimensional variables, tensors, and sparse allocation, along with a task-based driver for dependency-aware execution.
• A modular package system allows decoupling of physics solvers (e.g., hydrodynamics, radiation transport) for reuse and extensibility.

Experimental results

Research questions

• RQ1 Can a performance-portable AMR framework achieve high parallel efficiency across diverse exascale architectures, including GPUs and multiple CPU types?
• RQ2 How effectively can device memory allocation and logical packing of variables reduce kernel launch overhead in block-structured AMR simulations?
• RQ3 To what extent can one-sided, asynchronous MPI reduce communication latency and improve overlap with computation in multi-node AMR simulations?
• RQ4 Can the framework maintain high weak scaling efficiency at exascale scales, particularly on emerging architectures like Frontier?
• RQ5 How does the modular package abstraction support rapid prototyping and reuse in multi-physics applications?

Key findings

• Parthenon achieved 1.7×10¹³ zone-cycles/s on 9,216 Frontier nodes (73,728 logical GPUs), demonstrating exascale-level performance.
• The framework achieved approximately 92% weak scaling efficiency when scaling from a single node to 9,216 nodes on the Frontier supercomputer.
• Performance portability was successfully demonstrated across AMD and NVIDIA GPUs, Intel and AMD x86 CPUs, IBM Power9 CPUs, and Fujitsu A64FX CPUs.
• GPU-accelerated simulations were significantly faster than CPU-only simulations on the same node, both for uniform and multilevel grids with refinement.
• The Parthenon-hydro miniapp implemented full 3D compressible hydrodynamics with AMR in under 1,000 lines of code, showcasing rapid prototyping capability.
• The framework’s design enables efficient use of memory and communication, with reduced kernel launch overhead through logical packing of variables and blocks.