[Paper Review] The Plasma Simulation Code: A modern particle-in-cell code with load-balancing and GPU support

K. Germaschewski, W. Fox+6 more|arXiv (Cornell University)|Oct 29, 2013

Lightning and Electromagnetic Phenomena36 references16 citations

TL;DR

This paper presents the Plasma Simulation Code (psc), a modern, modular particle-in-cell (PIC) code with dynamic patch-based load balancing using space-filling curves and native GPU acceleration via CUDA. It achieves over 6× speedup on Cray XK7 systems by efficiently distributing computational work and leveraging GPU parallelism, enabling high-resolution kinetic plasma simulations on exascale-ready architectures.

ABSTRACT

Recent increases in supercomputing power, driven by the multi-core revolution and accelerators such as the IBM Cell processor, graphics processing units (GPUs) and Intel's Many Integrated Core (MIC) technology have enabled kinetic simulations of plasmas at unprecedented resolutions, but changing HPC architectures also come with challenges for writing efficient numerical codes. This paper describes the Plasma Simulation Code (PSC), an explicit, electromagnetic particle-in-cell code with support for different order particle shape functions. We focus on two distinguishing feature of the code: patch-based load balancing using space-filling curves, and support for Nvidia GPUs, which achieves substantial speed-up of up to more than 6x on the Cray XK7 architecture compared to a CPU-only implementation.

Motivation & Objective

• Address performance bottlenecks in large-scale kinetic plasma simulations caused by uneven particle distribution across computational domains.
• Overcome the challenges of evolving HPC architectures—especially multi-core and GPU accelerators—by designing a portable, modular PIC code.
• Enable efficient, scalable simulations on 10,000+ cores and thousands of GPUs by integrating advanced load-balancing and hardware-specific optimizations.
• Support both production-level science simulations and experimental development of new algorithms through a flexible, extensible codebase.
• Facilitate future portability to emerging architectures like Intel Xeon Phi by using modular, hardware-specific kernels with potential migration to higher-level abstractions like OpenACC.

Proposed method

• Implement a patch-based dynamic load-balancing strategy using space-filling curves to map 3D spatial patches to process ranks, minimizing load imbalance during particle migration.
• Use a capability-aware load-balancing algorithm that accounts for heterogeneous compute resources by assigning patches based on each process’s computational capacity.
• Integrate CUDA-based GPU kernels for particle push, field solve, and current deposition, enabling massive parallelism on Nvidia K20X GPUs.
• Decompose the simulation domain into spatial patches that are dynamically redistributed at runtime to maintain load balance as particles move across the grid.
• Support multiple particle shape functions (e.g., linear, quadratic) to improve numerical accuracy while maintaining computational efficiency.
• Design the code with modular C++ components to allow easy integration of new algorithms and hardware backends, such as Intel MIC or future accelerators.

Experimental results

Research questions

• RQ1 How can dynamic load balancing be effectively implemented in particle-in-cell simulations with highly mobile particles to maintain high performance?
• RQ2 To what extent can GPU acceleration improve the performance of electromagnetic PIC simulations on modern supercomputers?
• RQ3 Can a modular, extensible codebase efficiently support both production simulations and experimental algorithm development across diverse HPC architectures?
• RQ4 How does patch-based load balancing using space-filling curves compare to traditional domain decomposition in terms of load balance and scalability?
• RQ5 What performance gains are achievable by combining GPU acceleration with intelligent load balancing in large-scale plasma simulations?

Key findings

• The patch-based load-balancing algorithm using space-filling curves maintains high computational efficiency throughout simulations, even when particles migrate significantly across the domain.
• The Plasma Simulation Code achieves a sustained speed-up of over 6× on Cray XK7 nodes when using Nvidia K20X GPUs compared to CPU-only execution.
• The load-balancing strategy effectively handles dynamic workloads by redistributing patches based on real-time load measurements and process capabilities.
• The modular design of psc enables seamless integration of GPU kernels and supports future migration to other accelerators like Intel Xeon Phi.
• The code demonstrates strong scalability on systems with 10,000s of CPU cores and thousands of GPUs, making it suitable for exascale-class plasma simulations.
• The combination of dynamic load balancing and GPU acceleration enables high-resolution, multi-scale kinetic simulations spanning electron, ion, and global scales.