[Paper Review] Hierarchically deflated conjugate gradient

P A Boyle|arXiv (Cornell University)|Feb 11, 2014

Matrix Theory and Algorithms20 references16 citations

TL;DR

This paper introduces a hierarchically deflated conjugate gradient (HDCG) algorithm for solving five-dimensional chiral fermion systems, such as domain wall and Möbius fermions, on lattice QCD. By employing multi-level deflation with a four-dimensional coarse grid and an infra-red shift preconditioner, it reduces the stencil complexity and communication cost, achieving up to a 10× speedup in coarse grid operations and maintaining convergence efficiency on BlueGene/Q hardware.

ABSTRACT

We present a multi-level algorithm for the solution of five dimensional chiral fermion formulations, including domain wall and Mobius Fermions. The algorithm operates on the red-black preconditioned Hermitian operator, and directly accelerates conjugate gradients on the normal equations. The coarse grid representation of this matrix is next-to-next-to-next-to-nearest neighbour and multiple algorithmic advances are introduced, which help minimise the overhead of the coarse grid. The treatment of the coarse grids is purely four dimensional, and the bulk of the coarse grid operations are nearest neighbour. The intrinsic cost of most of the coarse grid operations is therefore comparable to those for the Wilson case. We also document the implementation of this algorithm in the BAGEL/Bfm software package and report on the measured performance gains the algorithm brings to simulations at the physical point on IBM BlueGene/Q hardware.

Motivation & Objective

• Address the slow convergence of conjugate gradient solvers in five-dimensional chiral fermion formulations due to high condition numbers from near-zero modes.
• Develop a scalable, multi-level deflation method that reduces the computational cost of coarse grid operations in 5D Dirac operators.
• Minimize algorithmic overhead in hierarchical multigrid solvers by using a four-dimensional coarse grid representation with nearest-neighbor stencils.
• Improve performance on BlueGene/Q by reducing communication bandwidth through low-precision data representation without sacrificing convergence.

Proposed method

• Apply red-black preconditioning to the Hermitian form of the 5D Dirac operator and solve the normal equations using conjugate gradients.
• Introduce a hierarchical deflation framework using a 1⁴ third-grid coarse space to represent the low-energy modes of the system.
• Implement an infra-red shift preconditioner based on truncated nearest-neighbor approximations of the little Dirac operator to reduce stencil complexity.
• Use ADEF-2 algorithm to improve robustness and reduce overhead in coarse grid solves by a factor of ten compared to standard Schur complement methods.
• Apply reduced-precision communication (e.g., 16-bit mantissa) in inter-node data exchange, preserving convergence while reducing bandwidth by up to fourfold.
• Maintain cache-friendly memory access patterns by reusing gauge and fermion fields across multiple matrix-vector operations.

Experimental results

Research questions

• RQ1 Can multi-level deflation be effectively applied to five-dimensional chiral fermion formulations like domain wall fermions without incurring prohibitive coarse grid costs?
• RQ2 How can the stencil complexity of the coarse grid operator be reduced to match that of standard Wilson fermions while preserving accuracy?
• RQ3 To what extent can communication bandwidth be reduced via low-precision data representation without degrading convergence in Krylov solvers?
• RQ4 Does the use of an infra-red shift preconditioner based on nearest-neighbor truncation improve performance and stability in 5D solvers?
• RQ5 Can the hierarchical deflation framework maintain robustness and efficiency in large-scale lattice QCD simulations on BlueGene/Q?

Key findings

• The HDCG algorithm achieved a 10× reduction in little Dirac operator overhead by using the ADEF-2 algorithm instead of the standard Schur complement method.
• The infra-red shift preconditioner reduced the bulk of coarse grid matrix multiplies to a nearest-neighbor stencil, matching the complexity of Wilson fermion solvers.
• Reduced-precision communication (7-bit mantissa) achieved a fourfold reduction in bandwidth with no loss in convergence, maintaining 168 outer iterations.
• The coarse grid operations were purely four-dimensional, enabling efficient reuse of gauge and fermion fields and improving cache locality.
• The algorithm achieved significant performance gains on IBM BlueGene/Q, with communication reduction outpacing gains from domain decomposition due to slower interconnect scaling.
• The preconditioned conjugate gradient solver remained robust to preconditioner variability, reducing the need for flexible Krylov methods.