[Paper Review] Numerically Stable Variants of the Communication-hiding Pipelined Conjugate Gradients Algorithm for the Parallel Solution of Large Scale Symmetric Linear Systems

Siegfried Cools, Wim Vanroose|arXiv (Cornell University)|Jun 19, 2017

Matrix Theory and Algorithms36 references3 citations

TL;DR

This paper introduces a numerically stable variant of the communication-hiding pipelined conjugate gradient (p-CG) method for solving large-scale symmetric positive definite linear systems. By reformulating key auxiliary variable recurrences using a shifted matrix $(A - \sigma I)$, the method suppresses the amplification of local rounding errors, restoring accuracy comparable to classical CG while maintaining high parallel scalability through reduced synchronization bottlenecks.

ABSTRACT

By reducing the number of global synchronization bottlenecks per iteration and hiding communication behind useful computational work, pipelined Krylov subspace methods achieve significantly improved parallel scalability on present-day HPC hardware. However, this typically comes at the cost of a reduced maximal attainable accuracy. This paper presents and compares several stabilized versions of the communication-hiding pipelined Conjugate Gradients method. The main novel contribution of this work is the reformulation of the multi-term recurrence pipelined CG algorithm by introducing shifts in the recursions for specific auxiliary variables. These shifts reduce the amplification of local rounding errors on the residual. The stability analysis presented in this work provides a rigorous method for selection of the optimal shift value in practice. It is shown that, given a proper choice for the shift parameter, the resulting shifted pipelined CG algorithm restores the attainable accuracy and displays nearly identical robustness to local rounding error propagation compared to classical CG. Numerical results on a variety of SPD benchmark problems compare different stabilization techniques for the pipelined CG algorithm, showing that the shifted pipelined CG algorithm is able to attain a high accuracy while displaying excellent parallel performance.

Motivation & Objective

• To address the reduced attainable accuracy in communication-hiding pipelined CG methods due to amplified local rounding errors.
• To develop a numerically stable variant of pipelined CG that matches the accuracy of classical CG in finite precision arithmetic.
• To provide a rigorous framework for selecting the optimal shift parameter $\sigma$ to control error propagation in multi-term recurrence pipelined CG.
• To demonstrate that the stabilized method maintains high parallel performance without introducing artificial computational overhead beyond minimal axpy operations.

Proposed method

• Reformulate the multi-term recurrence for auxiliary variables in pipelined CG by replacing the system matrix $A$ with a shifted matrix $(A - \sigma I)$ in the recurrence relations.
• Introduce a shift parameter $\sigma$ to control the amplification of local rounding errors on the residual and solution vectors.
• Derive a stability analysis based on local rounding error propagation matrices to guide the selection of the optimal $\sigma$ value.
• Use a posteriori estimation of error propagation matrices—computed from dot-product values $\alpha_i$ and $\beta_i$—to determine $\sigma$ in practice.
• Implement the shifted pipelined CG algorithm with minimal additional operations: only extra axpy steps, avoiding costly residual replacements or iterative refinement.
• Ensure equivalence to classical CG in exact arithmetic while preserving communication-hiding properties and pipelining benefits in finite precision.

Experimental results

Research questions

• RQ1 Can the loss of attainable accuracy in pipelined CG be mitigated without sacrificing communication hiding or parallel scalability?
• RQ2 How does the choice of shift parameter $\sigma$ affect the propagation of local rounding errors in the pipelined CG algorithm?
• RQ3 Is there a systematic method to select the optimal $\sigma$ value that restores accuracy comparable to classical CG?
• RQ4 Does the shifted pipelined CG method maintain the same robustness to rounding error propagation as classical CG?

Key findings

• The shifted pipelined CG method achieves a residual norm of $9.6 \times 10^{-12}$ on a 2D Poisson problem, matching classical CG's accuracy of $9.4 \times 10^{-12}$.
• The method attains this high accuracy in approximately 2.7 seconds on 20 nodes, achieving a $4\times$ speedup over classical CG.
• Classic CG stops scaling effectively beyond 4 nodes in strong scaling experiments, while pipelined variants—including the shifted version—scale to 20 nodes with a speedup of ~8×.
• The stabilized p-CG-rr and p-CG-sh variants show only a slight performance degradation compared to standard p-CG due to additional axpy operations, but maintain high accuracy.
• The stability analysis provides a rigorous method to select $\sigma$ based on error propagation matrices derived from $\alpha_i$ and $\beta_i$ values.
• Numerical experiments confirm that with optimal $\sigma$, the shifted pipelined CG achieves nearly identical robustness to rounding error propagation as classical CG.