[Paper Review] Numerical analysis of the maximal attainable accuracy in communication hiding pipelined Conjugate Gradient methods

Siegfried Cools|arXiv (Cornell University)|Apr 9, 2018

Matrix Theory and Algorithms34 references3 citations

TL;DR

This paper analyzes the maximal attainable accuracy in pipelined Conjugate Gradient (CG) methods, which reduce communication latency in HPC by overlapping global communication with local computations. It derives error propagation bounds for multi-term recurrence relations, showing that p(ℓ)-CG with ℓ ≥ 3 can achieve backward errors ≤ 1e-12, while lower-ℓ variants fail due to rounding error amplification.

ABSTRACT

Krylov subspace methods are widely known as efficient algebraic methods for solving large scale linear systems. However, on massively parallel hardware the performance of these methods is typically limited by communication latency rather than floating point performance. With HPC hardware advancing towards the exascale regime the gap between computation and communication keeps steadily increasing, imposing the need for scalable alternatives to traditional Krylov subspace methods. One such approach are the so-called pipelined Krylov subspace methods, which reduce the number of global synchronization points and overlap global communication latency with local arithmetic operations, thus hiding the global reduction phases behind useful computations. To obtain this overlap the traditional Krylov subspace algorithm is reformulated by introducing a number of auxiliary vector quantities, which are computed using additional recurrence relations. Although pipelined Krylov subspace methods are equivalent to traditional Krylov subspace methods in exact arithmetic, local rounding errors induced by the multi-term recurrence relations in finite precision may in practice affect convergence significantly. This numerical stability study aims to characterize the effect of local rounding errors on attainable accuracy in various pipelined versions of the popular Conjugate Gradient method. Expressions for the gaps between the true and recursively computed variables that are used to update the search directions in the different CG variants are derived. Furthermore, it is shown how these results can be used to analyze and correct the effect of local rounding error propagation on the maximal attainable accuracy of pipelined CG methods. The analysis in this work is supplemented by numerical experiments that demonstrate the numerical behavior of the pipelined CG methods.

Motivation & Objective

• To investigate the impact of local rounding errors on the maximal attainable accuracy in pipelined Krylov subspace methods.
• To analyze how multi-term recurrence relations in pipelined CG (p-CG and p(ℓ)-CG) amplify rounding errors in finite precision arithmetic.
• To derive practical bounds on error propagation that depend on pipelining depth ℓ and Krylov basis structure.
• To propose a countermeasure to reduce error amplification at the cost of computational efficiency.
• To demonstrate the numerical behavior of pipelined CG variants through reproducible experiments in PETSc.

Proposed method

• Derives expressions for the gaps between true and recursively computed variables in p-CG and p(ℓ)-CG, focusing on search direction updates.
• Uses finite precision analysis to model how local rounding errors propagate through multi-term recurrence relations in pipelined algorithms.
• Establishes theoretical bounds on error amplification as a function of the pipelining depth ℓ and the condition number of the Krylov basis.
• Proposes a stabilization strategy that trades computational cost for improved accuracy by modifying recurrence relations.
• Validates findings using numerical experiments with a C implementation in PETSc, comparing convergence and residual reduction across variants.
• Focuses on backward error as a metric for maximal attainable accuracy, particularly in the context of exascale HPC workloads.

Experimental results

Research questions

• RQ1 How do local rounding errors from multi-term recurrence relations affect the maximal attainable accuracy in pipelined CG methods?
• RQ2 What is the influence of pipelining depth ℓ on error amplification and convergence behavior in p(ℓ)-CG?
• RQ3 Why do p(1)-CG and p(2)-CG fail to achieve backward errors below 1e-12, while p(3)-CG succeeds?
• RQ4 Can a stabilization technique be designed to mitigate error amplification without sacrificing too much performance?
• RQ5 How does the structure of the Krylov basis affect the propagation of rounding errors in pipelined CG?

Key findings

• The p(ℓ)-CG method with ℓ = 3 achieves a backward error of ‖b - A𝐱̄ⱼ‖ / ‖b‖ ≤ 1.0e-12, while p(1)-CG and p(2)-CG do not, due to error amplification.
• The square root breakdown and subsequent restart in p(3)-CG improve final accuracy but delay convergence compared to lower-ℓ variants.
• Standard p(ℓ)-CG variants converge faster than their stabilized counterparts, but at the cost of reduced attainable accuracy.
• Error propagation is significantly influenced by the pipelining depth ℓ and the conditioning of the Krylov basis, with higher ℓ leading to better error control.
• A proposed countermeasure to reduce error amplification trades computational efficiency for improved accuracy, suggesting a viable trade-off for high-precision applications.
• The analysis reveals that p(ℓ)-CG with ℓ ≥ 3 can achieve higher maximal attainable accuracy than lower-ℓ variants due to better numerical stability in recurrence relations.