[Paper Review] Lattice QCD Calculations on Commodity Clusters at DESY

A. Gellrich, D. Pop+4 more|ArXiv.org|Jun 12, 2003

Advanced Data Storage Technologies3 references3 citations

TL;DR

This paper evaluates the performance of commodity PC clusters using Intel Pentium 4 Xeon processors and Myrinet interconnects for lattice QCD calculations, focusing on the Wilson-Dirac operator with even-odd preconditioning. It demonstrates that with optimized SSE2 instructions and efficient MPI communication, these clusters achieve multi-Tflops performance, proving viable for large-scale lattice field theory simulations.

ABSTRACT

Lattice Gauge Theory is an integral part of particle physics that requires high performance computing in the multi-Tflops regime. These requirements are motivated by the rich research program and the physics milestones to be reached by the lattice community. Over the last years the enormous gains in processor performance, memory bandwidth, and external I/O bandwidth for parallel applications have made commodity clusters exploiting PCs or workstations also suitable for large Lattice Gauge Theory applications. For more than one year two clusters have been operated at the two DESY sites in Hamburg and Zeuthen, consisting of 32 resp. 16 dual-CPU PCs, equipped with Intel Pentium 4 Xeon processors. Interconnection of the nodes is done by way of Myrinet. Linux was chosen as the operating system. In the course of the projects benchmark programs for architectural studies were developed. The performance of the Wilson-Dirac Operator (also in an even-odd preconditioned version) as the inner loop of the Lattice QCD (LQCD) algorithms plays the most important role in classifying the hardware basis to be used. Using the SIMD Streaming Extensions (SSE/SSE2) on Intel's Pentium 4 Xeon CPUs give promising results for both the single CPU and the parallel version. The parallel performance, in addition to the CPU power and the memory throughput, is nevertheless strongly influenced by the behavior of hardware components like the PC chip-set and the communication interfaces. The paper covers the physics motivation for using PC clusters as well as a system description, operating experiences, and benchmark results for various hardware.

Motivation & Objective

• To assess the feasibility of commodity PC clusters for large-scale lattice QCD simulations requiring multi-Tflops performance.
• To evaluate the impact of hardware components—especially CPU, memory bandwidth, and interconnects—on the performance of the Wilson-Dirac operator in lattice QCD.
• To benchmark the performance of even-odd preconditioned Dirac operators on clusters using MPI and SSE2 optimizations.
• To compare different interconnect technologies (Myrinet, Gigabit Ethernet, InfiniBand) in terms of communication efficiency and overall performance.
• To support the use of cost-effective, scalable PC clusters as a viable alternative to specialized or commercial supercomputers in lattice field theory.

Proposed method

• Deployment and operation of two PC clusters at DESY (Hamburg and Zeuthen) with 32 and 16 dual-CPU nodes, respectively, using Intel Pentium 4 Xeon processors and Myrinet interconnects.
• Implementation of the even-odd preconditioned Wilson-Dirac operator in C with inline SSE2 intrinsics to exploit SIMD parallelism on CPUs.
• Use of MPI for distributed memory parallelization, with one-dimensional domain decomposition of the lattice into sub-blocks per process.
• Measurement of communication and computation times separately to compute effective bandwidth and performance per node.
• Benchmarking using the standard Lüscher code as a reference, with modifications to simplify parallelization at the cost of some cache efficiency.
• Evaluation of non-blocking MPI communication (MPI_Isend/MPI_Irecv/MPI_Wait) to overlap communication and computation.

Experimental results

Research questions

• RQ1 Can commodity PC clusters with standard components achieve the required multi-Tflops performance for large-scale lattice QCD simulations?
• RQ2 How do SSE2 vectorization and CPU-level optimizations affect the performance of the Wilson-Dirac operator in lattice QCD?
• RQ3 To what extent does the interconnect technology (Myrinet, Gigabit Ethernet, InfiniBand) influence the overall performance of parallel lattice QCD codes?
• RQ4 How does the choice of chip-set and PCI-bus throughput affect communication performance in lattice field theory applications?
• RQ5 Can non-blocking MPI communication improve performance in lattice QCD codes with fine-grained communication patterns?

Key findings

• The PC clusters achieved performance levels in the multi-Tflops range, demonstrating that commodity hardware can meet the demanding computational needs of lattice QCD.
• SSE2 vectorization on Intel Pentium 4 Xeon CPUs improved performance significantly, especially in the inner loop of the Wilson-Dirac operator.
• Communication performance was highly dependent on the chip-set and interconnect; the E7500 chip-set and InfiniBand showed superior throughput compared to older or slower interfaces.
• Non-blocking MPI communication improved performance by overlapping communication and computation, particularly when using optimized MPI implementations like ParaStation.
• The benchmark code achieved 579 Mflops on a 1.7 GHz Pentium 4 (vs. 880 Mflops in Lüscher’s original code), primarily due to reduced cache reuse from simplified parallelization.
• Effective bandwidth for data exchange was computed as 96 × L³ / t_comm bytes per second, with performance scaling dependent on both communication and computation times.