[Paper Review] MILC Code Performance on High End CPU and GPU Supercomputer Clusters

Ruizi Li, DeTar, Carleton+2 more|arXiv (Cornell University)|Dec 1, 2017

Physics of Superconductivity and Magnetism3 references3 citations

TL;DR

This paper presents performance optimizations of the MILC lattice QCD code on high-end CPU and GPU clusters using QPhiX, QOPQDP, and QUDA libraries. It achieves up to 10x speedup in gauge force calculations on Intel Knights Landing by leveraging high arithmetic intensity and vectorized kernels, with strong weak scaling efficiency exceeding 80% on 2048-node clusters.

ABSTRACT

With recent developments in parallel supercomputing architecture, many core, multi-core, and GPU processors are now commonplace, resulting in more levels of parallelism, memory hierarchy, and programming complexity. It has been necessary to adapt the MILC code to these new processors starting with NVIDIA GPUs, and more recently, the Intel Xeon Phi processors. We report on our efforts to port and optimize our code for the Intel Knights Landing architecture. We consider performance of the MILC code with MPI and OpenMP, and optimizations with QOPQDP and QPhiX. For the latter approach, we concentrate on the staggered conjugate gradient and gauge force. We also consider performance on recent NVIDIA GPUs using the QUDA library.

Motivation & Objective

• To adapt the MILC lattice QCD code for modern high-performance computing architectures, including Intel Knights Landing (KNL) and NVIDIA GPUs.
• To address increasing programming complexity from multi-core, many-core, and GPU-based systems by enhancing parallelism and memory hierarchy utilization.
• To improve performance of key lattice QCD routines—conjugate gradient solvers, gauge forces, and fermion forces—through specialized libraries.
• To evaluate weak scaling efficiency and performance gains across KNL clusters and GPU nodes using optimized libraries.
• To enable future portability and performance portability across emerging architectures via library abstraction and vectorization.

Proposed method

• Ported and optimized the MILC code for Intel Knights Landing (KNL) using the QPhiX library, which supports 512-bit SIMD vector units and OpenMP threading.
• Employed the QOPQDP library for high-level optimizations of conjugate gradient and gauge force routines, leveraging data-parallel linear algebra and I/O routines.
• Used the QUDA library for GPU acceleration on NVIDIA P100 and K20 GPUs, enabling mixed-precision calculations and efficient gauge field compression.
• Applied structure-of-arrays (SOA) data layout in QPhiX for improved cache reuse, aligned with the Grid library's memory model.
• Optimized communication patterns in the gauge force routine to reduce latency and improve weak scaling efficiency.
• Benchmarked performance across multiple supercomputing centers (ALCF, NERSC, TACC) using the su3_rhmd_hisq application with $16^4$ and $24^4$ lattices.

Experimental results

Research questions

• RQ1 How does the performance of the MILC code's staggered conjugate gradient solver scale on Intel Knights Landing with QPhiX and QOPQDP optimizations?
• RQ2 What is the impact of high arithmetic intensity and vectorization on the weak scaling efficiency of the Symanzik one-loop gauge force on KNL clusters?
• RQ3 How do the performance characteristics of QPhiX and QUDA compare for gauge force and fermion force routines on KNL and GPU architectures?
• RQ4 To what extent does the use of optimized libraries (QPhiX, QOPQDP, QUDA) reduce runtime and improve scaling efficiency compared to the baseline MILC code?
• RQ5 How does hyper-threading and MPI/OpenMP thread configuration affect performance and scaling on KNL and GPU systems?

Key findings

• The QPhiX-optimized staggered conjugate gradient solver achieved a 1.5× performance improvement on a single KNL node compared to baseline MILC, though performance was limited by network bandwidth on clusters.
• The Symanzik one-loop gauge force routine with QPhiX achieved a 10× reduction in runtime compared to baseline MILC, with arithmetic intensity rising from 1.1 to over 2.0.
• Weak scaling of the gauge force on Cori II (2048 nodes) achieved over 80% efficiency with QPhiX, significantly higher than the 40% efficiency with baseline code.
• On a single GPU node, the P100 delivered approximately 2.3× higher performance than a KNL node for the gauge force, and the P100 outperformed the K20 by up to 5×.
• The QOPQDP-optimized HISQ fermion force routine reduced runtime to about 20% of baseline MILC, with weak scaling efficiency of 50–80% on up to 64 KNL nodes.
• Data remapping overhead currently matches the time of the gauge force routine, but is expected to be reduced once full integration of the algorithm into QPhiX is completed.