[Paper Review] High-performance orbit-following code ASCOT5 for Monte Carlo simulations in fusion plasmas

J. Varje, K. Särkimäki+6 more|arXiv (Cornell University)|Aug 7, 2019

Magnetic confinement fusion research13 references22 citations

TL;DR

ASCOT5 is a high-performance, orbit-following Monte Carlo code for simulating fast ion dynamics in fusion plasmas, redesigned from the ground up with a hybrid OpenMP-MPI parallelization and SIMD vectorization to exploit modern many-core CPUs. It achieves up to 6× speedup over ASCOT4 and accurately reproduces neoclassical transport and 3D plasma simulations with improved memory efficiency and scalability.

ABSTRACT

We present a novel implementation of a Monte Carlo particle-following code for solving the distribution function of minority species in fusion plasmas, called ASCOT5, and verify it using theoretical results for neoclassical transport. The code has been developed from ground up with an OpenMP-MPI hybrid paradigm to take full advantage of current and next generation many-core CPUs with multithreading and SIMD operations. Up to 6-fold increase in performance is demonstrated compared to a previous version of the code which only utilizes MPI. The physics model of the code is comprehensively validated against existing theoretical work, and it is shown to faithfully reproduce neoclassical diffusion across three different collisionality regimes. In simulations for realistic tokamak plasmas, including complex non-axisymmetric geometry, ASCOT5 is verified to reproduce results from the previous version ASCOT4.

Motivation & Objective

• Address the performance bottleneck of legacy ASCOT4 on modern many-core, SIMD-enabled CPUs by redesigning the codebase for full hardware utilization.
• Enable larger-scale simulations with higher statistical precision by improving computational efficiency and memory usage.
• Ensure accurate reproduction of neoclassical transport and realistic tokamak plasma behavior across varying collisionality regimes.
• Achieve near-ideal strong scaling in large-scale simulations using hybrid parallelization (OpenMP + MPI).
• Design a modular, maintainable codebase to support future extensions for MHD perturbations, time-dependent fields, and advanced physics models.

Proposed method

• Rewrote ASCOT5 in C using a hybrid OpenMP-MPI parallelization model to exploit multithreading and SIMD vector operations on modern CPUs.
• Explicitly structured the core simulation loop for efficient SIMD execution using 512-bit AVX-512 instructions, processing 8 markers simultaneously.
• Implemented a modular software architecture with clear interfaces between physics modules to enhance long-term maintainability.
• Employed a marker-based Monte Carlo method to solve the Fokker-Planck equation via the Langevin equation for charged particle trajectories.
• Optimized random number generation by replacing memory-heavy RNGs (e.g., SIMD Mersenne Twister) with more efficient algorithms to reduce performance overhead.
• Used a hybrid parallelization strategy where MPI handles inter-node communication and OpenMP manages multithreading within nodes, enabling efficient load distribution.

Experimental results

Research questions

• RQ1 Can a fully rewritten, SIMD-optimized Monte Carlo code achieve significant performance gains over legacy MPI-only implementations on modern many-core CPUs?
• RQ2 Does the new ASCOT5 code accurately reproduce theoretical neoclassical transport results across different collisionality regimes?
• RQ3 To what extent does ASCOT5 maintain fidelity to previous simulation results (e.g., ASCOT4) in complex 3D tokamak geometries?
• RQ4 How does the hybrid OpenMP-MPI parallelization strategy scale in large-scale simulations, particularly in terms of strong scaling?
• RQ5 Can the new codebase support higher marker counts and improved statistical precision within the same computational budget?

Key findings

• ASCOT5 achieves up to a 6-fold performance improvement over ASCOT4 in 2D JET-like simulations on Skylake CPUs, with a 4-fold speedup in 3D ITER-like cases.
• On Knights Landing systems, ASCOT5 achieves even greater speedup due to effective use of multithreading and SIMD, while ASCOT4 could not run due to memory limitations.
• The code reproduces theoretically predicted neoclassical diffusion across three collisionality regimes (banana, plateau, Pfirsch-Schlüter), validating its physics model.
• ASCOT5 successfully reproduces results from ASCOT4 in realistic 3D tokamak simulations, including non-axisymmetric magnetic perturbations.
• Strong scaling tests show near-ideal performance for simulations with 100,000 markers, with minimal overhead from data I/O and load imbalance.
• Optimizing random number generation by switching to a memory-efficient algorithm reduced performance bottlenecks and enabled full utilization of SIMD capabilities.