[Paper Review] Communication-optimal parallel and sequential QR and LU factorizations: theory and practice

James Demmel, Laura Grigori+2 more|ArXiv.org|Jun 12, 2008

graph theory and CDMA systemsEngineering50 references20 citations

TL;DR

This paper presents communication-optimal parallel and sequential QR and LU factorization algorithms—TSQR for tall and skinny matrices and CAQR for general matrices—using tree-based reduction and block Householder transformations to minimize data movement. The algorithms achieve optimal communication complexity (up to polylog factors) and outperform LAPACK/ScaLAPACK in both theory and practice, with speedups up to 22.9× on petascale models and 6.7× on clusters.

ABSTRACT

We present parallel and sequential dense QR factorization algorithms that are both optimal (up to polylogarithmic factors) in the amount of communication they perform, and just as stable as Householder QR. Our first algorithm, Tall Skinny QR (TSQR), factors m-by-n matrices in a one-dimensional (1-D) block cyclic row layout, and is optimized for m >> n. Our second algorithm, CAQR (Communication-Avoiding QR), factors general rectangular matrices distributed in a two-dimensional block cyclic layout. It invokes TSQR for each block column factorization.

Motivation & Objective

• Address the growing performance gap between floating-point operations and communication (latency/bandwidth) in high-performance computing.
• Develop algorithms that minimize communication in both sequential and parallel settings for dense QR and LU factorizations.
• Achieve theoretical communication optimality (matching lower bounds up to polylog factors) while maintaining numerical stability comparable to Householder QR.
• Demonstrate practical superiority over existing LAPACK and ScaLAPACK implementations in real-world performance on clusters and out-of-core systems.

Proposed method

• Design Tall Skinny QR (TSQR) for matrices with m ≫ n using a binary or general tree-based reduction to compute QR factorization with minimal communication.
• Implement CAQR (Communication-Avoiding QR) for general rectangular matrices by applying TSQR to block columns in a 2D block-cyclic layout.
• Use Householder reflections with structured BLAS-3 operations to optimize local factorizations and reduce data movement.
• Apply reduction trees to combine local QR results across processors, ensuring communication costs match theoretical lower bounds.
• Model performance using machine models that account for latency, bandwidth, and memory hierarchy to predict speedups.
• Extend lower bounds from matrix multiplication to QR and LU factorizations using geometric and combinatorial arguments (e.g., Loomis-Whitney inequality).

Experimental results

Research questions

• RQ1 Can QR and LU factorizations be redesigned to minimize communication in both sequential and parallel settings?
• RQ2 What are the theoretical lower bounds on communication for QR and LU factorizations, and can they be achieved in practice?
• RQ3 How do communication-avoiding algorithms compare to existing LAPACK and ScaLAPACK implementations in terms of performance and numerical stability?
• RQ4 To what extent can communication-avoiding factorizations be scaled across distributed-memory systems and out-of-core main memory?
• RQ5 Can the communication-avoiding approach be generalized to both tall/skinny and general rectangular matrices with minimal loss in efficiency?

Key findings

• TSQR and CAQR achieve communication-optimality (up to polylogarithmic factors) for both sequential and parallel execution, matching theoretical lower bounds derived from matrix multiplication.
• On a 16-processor Pentium III cluster, parallel TSQR achieved speedups of up to 6.7× over ScaLAPACK’s PDGEQRF.
• On a 32-processor BlueGene/L system, TSQR achieved up to 4× speedup in parallel execution.
• Sequential out-of-DRAM TSQR on a laptop was only 2× slower than predicted with infinite DRAM, demonstrating strong performance on memory-constrained systems.
• Performance modeling predicted up to 22.9× speedup for CAQR over ScaLAPACK on a model petascale machine and up to 9.7× on an IBM Power5.
• CAQR and TSQR outperform LAPACK and ScaLAPACK in both latency and bandwidth in sequential and parallel cases, respectively, while maintaining numerical stability equivalent to Householder QR.