[Paper Review] True BLAS-3 Performance QRCP using Random Sampling

Jed A. Duersch, Ming Gu|arXiv (Cornell University)|Sep 23, 2015

Stochastic Gradient Optimization Techniques7 citations

TL;DR

This paper proposes a randomized, communication-avoiding QRCP algorithm using BLAS-3 operations and sample-based column norm approximation to achieve near-unpivoted QR performance while maintaining accuracy comparable to traditional QRCP. By avoiding costly trailing matrix updates and introducing a sample update formula, it enables fast, scalable truncated SVD and low-rank approximations with strong parallel performance on shared-memory systems.

ABSTRACT

The dominant contribution to communication complexity in factorizing a matrix using QR with column pivoting is due to column-norm updates that are required to process pivot decisions. We use randomized sampling to approximate this process which dramatically reduces communication in column selection. We also introduce a sample update formula to reduce the cost of sampling trailing matrices. Using our column selection mechanism we observe results that are comparable to those obtained from the QRCP algorithm, but with performance near unpivoted QR. We also demonstrate strong parallel scalability on shared memory multiple core systems using an implementation in Fortran with OpenMP. This work immediately extends to produce low-rank truncated approximations of large matrices. We propose a truncated QR factorization with column pivoting that avoids trailing matrix updates which are used in current implementations of BLAS-3 QR and QRCP. Provided the truncation rank is small, avoiding trailing matrix updates reduces approximation time by nearly half. By using these techniques and employing a variation on Stewart's QLP algorithm, we develop an approximate truncated SVD that runs nearly as fast as truncated QR.

Motivation & Objective

• To reduce communication overhead in QRCP factorization caused by frequent column-norm updates during pivot selection.
• To enable near-unpivoted QR performance while preserving the numerical stability and accuracy of column-pivoted QR.
• To accelerate truncated matrix factorizations by eliminating expensive trailing matrix updates in BLAS-3 QR and QRCP.
• To achieve strong parallel scalability on shared-memory multi-core architectures using OpenMP and Fortran.
• To develop an efficient truncated SVD via a modified QLP algorithm that leverages the proposed sampling-based QRCP.

Proposed method

• Uses randomized sampling to approximate column norms, replacing expensive sequential updates with batched, BLAS-3-optimized operations.
• Introduces a sample update formula to efficiently maintain sampling information during trailing matrix updates, reducing computational cost.
• Employs a modified version of Stewart’s QLP algorithm to construct an approximate truncated SVD with minimal overhead.
• Designs a truncated QR factorization that avoids full trailing matrix updates, significantly reducing computation time for low-rank approximations.
• Implements the algorithm in Fortran with OpenMP to achieve strong parallel scalability on shared-memory systems.
• Applies the method to produce low-rank matrix approximations with performance close to unpivoted QR but with QRCP-level accuracy.

Experimental results

Research questions

• RQ1 Can randomized sampling effectively approximate column norms in QRCP to reduce communication costs without sacrificing accuracy?
• RQ2 To what extent can trailing matrix updates be avoided in BLAS-3 QR and QRCP to accelerate truncated factorizations?
• RQ3 How does the proposed method compare in performance and accuracy to standard QRCP and unpivoted QR in low-rank approximation tasks?
• RQ4 Can the algorithm achieve strong parallel scalability on shared-memory multi-core systems while maintaining high performance?
• RQ5 What is the performance gain of the proposed truncated SVD over existing methods when the truncation rank is small?

Key findings

• The proposed method achieves performance near unpivoted QR while maintaining accuracy comparable to traditional QRCP, significantly reducing communication costs.
• Avoiding trailing matrix updates in truncated QR reduces approximation time by nearly half when the truncation rank is small.
• The sample update formula effectively reduces the cost of maintaining sampling information during factorization, enabling efficient computation.
• The implementation demonstrates strong parallel scalability on shared-memory multi-core systems using OpenMP and Fortran.
• The resulting approximate truncated SVD runs nearly as fast as truncated QR, enabling efficient low-rank matrix approximations.
• The method enables high-performance, communication-avoiding matrix factorizations suitable for large-scale data analysis.