[Paper Review] Low synchronization GMRES algorithms

Kasia Świrydowicz, Julien Langou+3 more|arXiv (Cornell University)|Sep 16, 2018

Matrix Theory and Algorithms19 references4 citations

TL;DR

This paper introduces low-synchronization GMRES algorithms based on modified Gram-Schmidt (MGS) and classical Gram-Schmidt (CGS) orthogonalization, using a lagged normalization in the compact WY form to reduce global reductions to just one per iteration. The method achieves backward stability with O(εκ(A)) loss of orthogonality and delivers up to 35× speedup on GPUs over Hypre's BLAS-1 MGS-GMRES, enabling scalable exascale performance with reduced communication overhead.

ABSTRACT

Communication-avoiding and pipelined variants of Krylov solvers are critical for the scalability of linear system solvers on future exascale architectures. We present low synchronization variants of iterated classical (CGS) and modified Gram-Schmidt (MGS) algorithms that require one and two global reduction communication steps. Derivations of low synchronization iterated CGS algorithms are based on previous work by Ruhe. Our main contribution is to introduce a backward normalization lag into the compact $WY$ form of MGS resulting in a ${\cal O}(\eps)κ(A)$ stable GMRES algorithm that requires only one global synchronization per iteration. The reduction operations are overlapped with computations and pipelined to optimize performance. Further improvements in performance are achieved by accelerating GMRES BLAS-2 operations on GPUs.

Motivation & Objective

• Address the communication bottleneck in Krylov solvers on exascale architectures by minimizing global synchronization steps.
• Develop backward-stable GMRES variants with reduced communication using compact WY and lagged normalization in MGS.
• Enable scalable performance on GPUs and distributed-memory systems by overlapping reductions with computation and pipelining.
• Achieve high performance and stability for large sparse linear systems, particularly in FEM and CFD applications.
• Extend the scalability of GMRES beyond current limits by reducing synchronization frequency while maintaining numerical accuracy.

Proposed method

• Introduce a lag in the diagonal normalization of the R matrix in compact WY MGS to delay global reductions, reducing synchronization to one per iteration.
• Leverage the loop-invariant formulation from Ruhe (1994) to unroll reorthogonalization and eliminate redundant global reductions in CGS.
• Apply pipelining and overlap of reduction operations with computation, inspired by Ghysels et al. and Yamazaki et al., to hide communication latency.
• Implement GPU-accelerated BLAS-2 operations for SpMV and inner products to enhance performance on modern accelerators.
• Use the compact WY representation to reduce storage and improve data locality in orthogonalization steps.
• Integrate the low-synchronization GMRES with preconditioners like BoomerAMG and L1 Jacobi smoothers on GPUs for full solver scalability.

Experimental results

Research questions

• RQ1 Can the number of global synchronization steps in MGS-GMRES be reduced to one per iteration without sacrificing backward stability?
• RQ2 How does lagged normalization in the compact WY form affect the loss of orthogonality and numerical stability in GMRES?
• RQ3 To what extent can pipelining and overlapping of reductions improve performance on GPU-accelerated exascale systems?
• RQ4 What is the performance gain of low-synchronization GMRES over traditional Hypre BLAS-1 MGS-GMRES on large sparse linear systems?
• RQ5 Can the proposed algorithm scale efficiently across O(100K) nodes with minimal communication overhead?

Key findings

• The one-synchronization MGS-GMRES algorithm achieves backward stability with O(εκ(A)) loss of orthogonality, ensuring numerical reliability.
• The new GMRES-two-synch algorithm reduces solve time to 1.92 seconds for a 500K system at 1e-13 tolerance, a 60% improvement over Hypre's BLAS-1 MGS-GMRES.
• GPU-accelerated implementations achieve up to 35× speedup compared to Hypre's Level-1 BLAS MGS-GMRES on the same hardware.
• Gram-Schmidt kernel time decreases significantly with increasing node count on the Peregrine supercomputer, demonstrating strong scaling.
• The method enables a shift toward more, cheaper GMRES iterations combined with lightweight V-cycles, optimizing total solve time.
• The approach shows promise for large-scale distributed-memory systems, with initial results indicating scalability at scale.