[Paper Review] An Efficient, Sparsity-Preserving, Online Algorithm for Low-Rank Approximation

David G. Anderson, Ming Gu|arXiv (Cornell University)|Feb 18, 2016

Sparse and Compressive Sensing Techniques35 references3 citations

TL;DR

This paper introduces Spectrum-Revealing LU (SRLU), a novel, sparsity-preserving, online algorithm for efficient low-rank matrix approximation. By combining randomized sampling with a deterministic refinement step, SRLU achieves high-accuracy low-rank approximations with computational complexity competitive with SVD, while preserving sparsity and enabling efficient online updates.

ABSTRACT

Low-rank matrix approximation is a fundamental tool in data analysis for processing large datasets, reducing noise, and finding important signals. In this work, we present a novel truncated LU factorization called Spectrum-Revealing LU (SRLU) for effective low-rank matrix approximation, and develop a fast algorithm to compute an SRLU factorization. We provide both matrix and singular value approximation error bounds for the SRLU approximation computed by our algorithm. Our analysis suggests that SRLU is competitive with the best low-rank matrix approximation methods, deterministic or randomized, in both computational complexity and approximation quality. Numeric experiments illustrate that SRLU preserves sparsity, highlights important data features and variables, can be efficiently updated, and calculates data approximations nearly as accurately as possible. To the best of our knowledge this is the first practical variant of the LU factorization for effective and efficient low-rank matrix approximation.

Motivation & Objective

• Address the need for efficient, scalable, and sparsity-preserving low-rank approximation in large-scale data analysis.
• Overcome the limitations of standard LU factorization, which lacks stability and spectrum-revealing properties in truncated form.
• Develop an online algorithm that supports incremental updates and identifies important data features and variables.
• Achieve approximation accuracy within a constant factor of the best possible low-rank approximation, regardless of matrix rank.
• Provide theoretical error bounds and empirical validation for both matrix and singular value approximation quality.

Proposed method

• Propose a truncated LU factorization called Spectrum-Revealing LU (SRLU), where permutation matrices Π₁ and Π₂ are used to reveal the spectrum of the matrix.
• Use randomized sampling to efficiently identify high-impact rows and columns for pivoting, enabling scalable computation.
• Apply a deterministic follow-up procedure after random sampling to ensure high-quality low-rank approximation.
• Integrate blocking and block-wise LU factorization (Crout-style) to optimize performance on modern architectures.
• Preserve sparsity through careful pivoting and matrix update strategies, minimizing fill-in during factorization.
• Support online updates by reweighting older data via scaling factors applied before spectrum-revealing pivoting.

Experimental results

Research questions

• RQ1 Can a truncated LU factorization be made both spectrum-revealing and computationally efficient for large-scale low-rank approximation?
• RQ2 How can complete pivoting be made practical in an online, incremental setting without requiring full matrix access at each step?
• RQ3 To what extent does the proposed SRLU algorithm preserve sparsity in sparse input matrices compared to standard LU or SVD?
• RQ4 What is the theoretical approximation error bound of SRLU relative to the optimal low-rank approximation?
• RQ5 How does SRLU’s performance compare to SVD and other randomized or deterministic low-rank methods in terms of accuracy and speed?

Key findings

• SRLU achieves low-rank approximation accuracy within a constant factor of the best possible approximation, matching the theoretical performance of truncated SVD.
• The algorithm preserves sparsity effectively: in experiments, SRLU maintained non-zero entries at 1,570 (oscil_dcop), 62.7K (g7jac020), 2.2K (tols1090), and 184K (mhd1280a), significantly lower than LU or SVD.
• SRLU relative error was as low as 1.03e-3 for oscil_dcop and 4.98e-6 for mhd1280a, demonstrating high approximation quality.
• TRLUCP (the core algorithm) is up to twice as fast as LAPACK’s DGETRF when using optimal block sizes, and outperforms standard right-looking LU in computation time.
• The algorithm supports online updates by reweighting older data, enabling dynamic data approximation with reduced importance for outdated observations.
• Theoretical analysis confirms that SRLU provides both matrix and singular value approximation error bounds, ensuring robustness and reliability.