[Paper Review] Hybrid Monte Carlo with Wilson Dirac operator on the Fermi GPU

Abhijit Chakrabarty, Pushan Majumdar|arXiv (Cornell University)|Jul 10, 2012

Theoretical and Computational Physics3 citations

TL;DR

This paper presents a highly optimized Hybrid Monte Carlo simulation for lattice QCD with two degenerate Wilson-Dirac fermions on NVIDIA Fermi GPUs, achieving over 10x speedup over a single CPU thread by mapping array variables to registers via code generation. The approach minimizes global memory access, leveraging GPU register files for near-instantaneous data access and achieving performance equivalent to 19–26 CPU threads on high-end clusters.

ABSTRACT

In this article we present our implementation of a Hybrid Monte Carlo algorithm for Lattice Gauge Theory using two degenerate flavours of Wilson-Dirac fermions on a Fermi GPU. We find that using registers instead of global memory speeds up the code by almost an order of magnitude. To map the array variables to scalars, so that the compiler puts them in the registers, we use code generators. Our final program is more than 10 times faster than a generic single CPU.

Motivation & Objective

• To accelerate Hybrid Monte Carlo simulations for lattice QCD with dynamical fermions on GPU architectures.
• To overcome the performance bottleneck caused by slow global memory access in GPU-based lattice gauge theory simulations.
• To explore the use of register variables instead of global memory for runtime data in GPU kernels.
• To develop and apply code generation techniques to map array variables to scalar registers for optimal performance.
• To demonstrate that register optimization can yield over 10x speedup over single-threaded CPU performance on realistic lattice sizes.

Proposed method

• Implement the Hybrid Monte Carlo algorithm for two degenerate Wilson-Dirac fermions on Fermi GPUs using CUDA.
• Replace global memory usage for runtime variables with register storage to reduce latency from ~100–150 cycles to 1 cycle.
• Use a code generator to map array elements to scalar variables so the compiler can assign them to registers.
• Optimize the conjugate gradient inversion kernel by minimizing global memory transactions and maximizing register utilization.
• Test performance on C2050 and X2090 Fermi GPUs and compare against CPU clusters (Opteron, Cray XE6) using Intel and Cray Fortran compilers.
• Measure timing across CG iterations to estimate kernel launch overhead and scaling behavior.

Experimental results

Research questions

• RQ1 Can register-based storage significantly outperform global memory access in GPU-based HMC simulations for lattice QCD?
• RQ2 To what extent can code generation techniques map array variables to registers to improve kernel performance?
• RQ3 How does the performance of register-optimized GPU kernels compare to multi-threaded CPU implementations on modern clusters?
• RQ4 What is the achievable speedup of a GPU-optimized HMC implementation over a single CPU thread for realistic lattice sizes?
• RQ5 Does the performance gain from register optimization scale with increasing lattice volume and iteration count?

Key findings

• Using registers instead of global memory accelerates the code by nearly an order of magnitude, achieving over 10x speedup compared to a single CPU thread.
• The performance of the C2050 GPU is equivalent to approximately 19 threads of a Cray XE6 node, while the X2090 matches about 26 such threads.
• The kernel launch overhead for the conjugate gradient routine is approximately 0.5 seconds on both C2050 and X2090 GPUs.
• The timing of the inverter scales linearly with the number of CG iterations, confirming consistent performance across iterations.
• Without register mapping, the performance gain was only about 2x; full register optimization increased this to over 10x.
• The use of code generation was essential to effectively map arrays to registers, enabling the compiler to place variables in high-speed register files.