[Paper Review] QPACE 2 and Domain Decomposition on the Intel Xeon Phi

Paul Arts, Jacques Bloch+16 more|arXiv (Cornell University)|Feb 13, 2015

Parallel Computing and Optimization Techniques8 references3 citations

TL;DR

This paper presents QPACE 2, a custom supercomputer based on Intel Xeon Phi processors designed for high-performance lattice QCD simulations. It implements a domain-decomposition-based solver that achieves near-linear strong scaling across 60 cores and improves time-to-solution by a factor of 5 in strong scaling and 2–3× in weak scaling compared to non-DD solvers, demonstrating superior scalability and efficiency on the KNC architecture.

ABSTRACT

We give an overview of QPACE 2, which is a custom-designed supercomputer based on Intel Xeon Phi processors, developed in a collaboration of Regensburg University and Eurotech. We give some general recommendations for how to write high-performance code for the Xeon Phi and then discuss our implementation of a domain-decomposition-based solver and present a number of benchmarks.

Motivation & Objective

• To design and deploy a cost- and energy-efficient supercomputer, QPACE 2, using Intel Xeon Phi (Knights Corner) processors for lattice QCD simulations.
• To address the limitations of bandwidth-bound solvers by implementing a domain-decomposition (DD) preconditioner that is more latency-tolerant and scalable.
• To optimize high-performance computing workloads for the Xeon Phi architecture, particularly focusing on memory access patterns and instruction-level parallelism.
• To evaluate the strong and weak scaling behavior of the DD solver on multi-node systems using the TACC Stampede cluster as a proxy for the full QPACE 2 system.
• To lay the groundwork for QPACE 3, an upgrade to the Knights Landing Xeon Phi architecture with improved performance and memory bandwidth.

Proposed method

• QPACE 2 uses a custom node design with four Intel Xeon Phi 7120X processors, a low-power CPU, and a dual-port FDR Infiniband card, all connected via a PCIe switch to enable peer-to-peer communication.
• The system employs a two-layer network topology based on PCIe and FDR Infiniband, enabling efficient inter-node communication with low latency and high bandwidth.
• A domain-decomposition-based solver is implemented to reduce memory bandwidth pressure and improve scalability, particularly in the context of the KNC’s memory hierarchy.
• Optimization techniques include L1 and L2 software prefetching, instruction-level tuning (e.g., fused multiply-add usage), and careful load balancing across 60 KNC cores.
• Performance is analyzed using Intel VTune for profiling, and key metrics such as Gflop/s/core and time-to-solution are measured across different lattice sizes and configurations.
• Multi-node benchmarks are conducted on the TACC Stampede cluster using the 7110P Xeon Phi variant to simulate the full QPACE 2 system’s behavior.

Experimental results

Research questions

• RQ1 How does the domain-decomposition preconditioner compare to non-DD solvers in terms of strong and weak scaling on the Xeon Phi architecture?
• RQ2 What performance bottlenecks limit single-core performance on the Xeon Phi 7120X, and how can they be mitigated through software optimization?
• RQ3 To what extent does the memory hierarchy and cache behavior of the KNC affect the performance of lattice QCD solvers?
• RQ4 How does load imbalance affect multi-core scaling when distributing domains across 60 KNC cores, and what strategies can minimize this effect?
• RQ5 What is the expected performance gain of the DD solver over non-DD solvers in realistic HMC and data analysis workloads?

Key findings

• The single-core performance of the MR iteration reaches 13.3 Gflop/s/core in single precision, with a performance limit of 22 Gflop/s/core due to instruction-level overheads and memory stalls.
• The overall domain-decomposition preconditioner achieves 9.5 Gflop/s/core, with performance loss primarily due to non-MR components and memory access patterns.
• The DD solver scales nearly linearly on 60 cores when the number of domains is sufficiently large, but load imbalance becomes significant when the number of domains approaches the number of cores.
• In strong scaling, the DD solver maintains linear scaling up to 1024 nodes (1024 KNCs) on the TACC Stampede cluster, achieving a 5× speedup over the non-DD solver.
• In weak scaling, the DD solver outperforms the non-DD solver by a factor of 2–3, demonstrating its suitability for large-scale data analysis.
• The system demonstrates stable operation over a month-long continuous benchmark, validating the reliability of the QPACE 2 architecture.