[Paper Review] High-speed turbulent flows towards the exascale: STREAmS-2 porting and performance

Srikanth Sathyanarayana, Matteo Bernardini+3 more|arXiv (Cornell University)|Apr 11, 2023

Lattice Boltzmann Simulation Studies4 citations

TL;DR

This paper presents STREAmS-2, a portable, high-performance compressible flow solver for exascale HPC systems, extending its CUDA Fortran backend with a new HIPFort backend for AMD GPUs. It achieves over 95% weak scalability on 512 nodes, enabling trillion-point simulations of supersonic and hypersonic turbulent flows.

ABSTRACT

Exascale High Performance Computing (HPC) represents a tremendous opportunity to push the boundaries of Computational Fluid Dynamics (CFD), but despite the consolidated trend towards the use of Graphics Processing Units (GPUs), programmability is still an issue. STREAmS-2 (Bernardini et al. Comput. Phys. Commun. 285 (2023) 108644) is a compressible solver for canonical wall-bounded turbulent flows capable of harvesting the potential of NVIDIA GPUs. Here we extend the already available CUDA Fortran backend with a novel HIPFort backend targeting AMD GPU architectures. The main implementation strategies are discussed along with a novel Python tool that can generate the HIPFort and CPU code versions allowing developers to focus their attention only on the CUDA Fortran backend. Single GPU performance is analysed focusing on NVIDIA A100 and AMD MI250x cards which are currently at the core of several HPC clusters. The gap between peak GPU performance and STREAmS-2 performance is found to be generally smaller for NVIDIA cards. Roofline analysis allows tracing this behavior to unexpectedly different computational intensities of the same kernel using the two cards. Parallel performance is measured on the two largest EuroHPC pre-exascale systems, LUMI (AMD GPUs) and Leonardo (NVIDIA GPUs). Strong scalability reveals more than 80% efficiency up to 16 nodes for Leonardo and up to 32 for LUMI. Weak scalability shows an impressive efficiency of over 95% up to the maximum number of nodes tested (256 for LUMI and 512 for Leonardo). This analysis shows that STREAmS-2 is the perfect candidate to fully exploit the power of current pre-exascale HPC systems in Europe, allowing users to simulate flows with over a trillion mesh points, thus reducing the gap between the Reynolds numbers achievable in high-fidelity simulations and those of real engineering applications.

Motivation & Objective

• Address the challenge of GPU portability in high-fidelity CFD simulations across heterogeneous HPC architectures.
• Extend the STREAmS-2 solver to support both NVIDIA and AMD GPU architectures while maintaining code readability and maintainability.
• Enable large-scale Direct Numerical Simulations (DNS) of compressible wall-bounded turbulent flows on pre-exascale systems.
• Optimize performance across different GPU architectures using roofline analysis and communication-avoiding patterns.
• Achieve near-ideal strong and weak scalability on major EuroHPC pre-exascale systems, LUMI and Leonardo.

Proposed method

• Developed a novel HIPFort backend for AMD GPU architectures, complementing the existing CUDA Fortran backend.
• Designed a Python-based code generation tool to automatically convert CUDA Fortran kernels into HIPFort and CPU code, ensuring consistency and reducing maintenance overhead.
• Employed an object-oriented, multi-backend software architecture to decouple algorithmic logic from hardware-specific implementations.
• Conducted single-GPU performance analysis on NVIDIA A100 and AMD MI250x GPUs, evaluating grid size, loop parallelization, thread masking, and divergence.
• Applied roofline performance modeling to identify differences in computational intensity between GPU architectures.
• Evaluated strong and weak scalability on LUMI (AMD) and Leonardo (NVIDIA) using both synchronous and asynchronous communication patterns.

Experimental results

Research questions

• RQ1 How does the performance of STREAmS-2 compare between NVIDIA A100 and AMD MI250x GPUs in single-node configurations?
• RQ2 What causes the performance gap between peak GPU bandwidth and actual STREAmS-2 performance on different GPU architectures?
• RQ3 To what extent does asynchronous communication improve strong and weak scalability on large-scale HPC systems?
• RQ4 Can the new HIPFort backend achieve comparable performance to CUDA Fortran on AMD-based systems like LUMI?
• RQ5 What is the maximum problem size (in grid points) that STREAmS-2 can efficiently simulate on current pre-exascale systems?

Key findings

• The performance gap between peak GPU bandwidth and actual STREAmS-2 performance is smaller on NVIDIA A100 than on AMD MI250x, despite similar theoretical peak performance.
• Roofline analysis reveals that the computational intensity of the same kernel is significantly different on NVIDIA and AMD GPUs, explaining the performance disparity.
• Strong scalability exceeds 80% efficiency up to 16 nodes on Leonardo (NVIDIA) and 32 nodes on LUMI (AMD), demonstrating excellent strong scaling.
• Weak scalability exceeds 95% efficiency up to 256 nodes on LUMI and 512 nodes on Leonardo, indicating near-ideal weak scaling.
• Asynchronous communication reduces performance loss by 1–2.5% compared to synchronous mode, with greater benefit at smaller grid sizes per GPU.
• The system can efficiently simulate flows with over a trillion mesh points, significantly reducing the gap between simulated and real-world Reynolds numbers.