[Paper Review] Alsvinn: A Fast multi-GPGPU finite volume solver with a strong emphasis on reproducibility

Kjetil Olsen Lye|arXiv (Cornell University)|Dec 16, 2019

Meteorological Phenomena and Simulations4 citations

TL;DR

Alsvinn is a high-performance, multi-GPGPU finite volume solver for hyperbolic conservation laws with built-in uncertainty quantification (UQ), enabling scalable, reproducible simulations of complex 3D flows. It achieves excellent weak scaling across 512 GPUs and over 60% memory bandwidth utilization on NVIDIA P100, demonstrating order-of-magnitude speedups over CPU-based solvers for UQ-intensive problems.

ABSTRACT

We present the Alsvinn simulator, a fast multi general purpose graphical processing unit (GPGPU) finite volume solver for hyperbolic conservation laws in multiple space dimensions. Alsvinn has native support for uncertainty quantifications, and exhibits excellent scaling on top tier compute clusters.

Motivation & Objective

• To address the computational bottleneck of uncertainty quantification (UQ) in hyperbolic conservation laws by enabling efficient, reproducible, and scalable simulations on modern GPGPU clusters.
• To develop a finite volume solver that natively supports multiple UQ methods—Monte Carlo, Quasi-Monte Carlo, and Multilevel Monte Carlo—within a single high-performance framework.
• To ensure high performance and strong scaling across multi-GPU and multi-node architectures by overlapping communication with computation and minimizing data movement.
• To provide a portable, extensible C++/CUDA/MPI implementation that supports a wide range of equations, numerical fluxes, and reconstruction schemes with minimal developer overhead.
• To demonstrate that probabilistic formulations of ill-posed hyperbolic systems can be efficiently simulated at high spatio-temporal and statistical resolution using GPGPU acceleration.

Proposed method

• Alsvinn implements a semi-discrete finite volume method using high-order reconstruction (e.g., WENO) and strong stability preserving Runge-Kutta time stepping for hyperbolic conservation laws.
• It employs a hybrid compile-time and run-time polymorphism model to generate optimized CPU and GPGPU kernels automatically, enabling domain-specific code without GPU programming expertise.
• The solver supports multiple numerical fluxes, including HLLC, Godunov, Roe, and entropy-conservative fluxes, with runtime selection and automatic kernel generation.
• Communication-avoiding and overlap techniques are used in multi-GPU setups, where halo exchanges are compactified and performed in parallel with inner-domain computation.
• The UQ module integrates statistical solution frameworks via sampling-based stochastic integration, supporting single-level and multilevel Monte Carlo methods with MPI load balancing.
• All UQ operations, including mean, variance, and PDF estimation, are computed directly on the GPU to minimize data transfer and storage overhead.

Experimental results

Research questions

• RQ1 Can a multi-GPGPU finite volume solver achieve strong weak scaling across 512 GPUs while maintaining high memory bandwidth utilization for 3D hyperbolic PDEs?
• RQ2 How does the performance of a GPGPU-based UQ-enabled finite volume solver compare to highly optimized CPU-based solvers for complex 3D flows?
• RQ3 To what extent can communication overhead be minimized in multi-GPU finite volume simulations through overlapping and compactification techniques?
• RQ4 Can a single software stack natively support both high-order numerical schemes and diverse UQ sampling methods with minimal performance penalty?
• RQ5 Is it feasible to perform high-resolution statistical solutions of ill-posed hyperbolic conservation laws using GPGPU acceleration at scale?

Key findings

• Alsvinn achieves excellent weak scaling across 512 GPUs for both 2D and 3D simulations of the Kelvin–Helmholtz instability, with runtime per timestep remaining nearly constant as resolution increases.
• The code achieves 0.105 seconds per timestep at a 3D resolution of 2048³, demonstrating high performance for large-scale UQ simulations.
• MPI communication overhead is less than 10% in 2D and around 30% in 3D when using 512 GPUs, which is expected due to the O(N²) halo data growth in 3D.
• On a single NVIDIA Tesla P100, Alsvinn achieves over 60% of the theoretical memory bandwidth, confirming it is memory bandwidth bound and well-optimized.
• The solver attains approximately 10% of peak FLOPS, consistent with memory-bound finite volume algorithms, and maintains high efficiency across complex, high-order schemes.
• Alsvinn enables the first global-scale simulations of statistical solutions for 3D compressible flows at high spatio-temporal and statistical resolution, demonstrating feasibility and performance at scale.