[Paper Review] Fast and stable randomized low-rank matrix approximation

Yuji Nakatsukasa|arXiv (Cornell University)|Sep 23, 2020

Sparse and Compressive Sensing Techniques55 references18 citations

TL;DR

This paper introduces generalized Nyström, a numerically stable, single-pass randomized low-rank matrix approximation method for general dense and sparse matrices. It achieves near-optimal complexity of $O(mn\log n + r^3)$, outperforming HMT and Tropp by up to 10× in speed while maintaining comparable accuracy, especially for large $r$, and enables efficient updating and downdating without orthogonalization.

ABSTRACT

Randomized SVD has become an extremely successful approach for efficiently computing a low-rank approximation of matrices. In particular the paper by Halko, Martinsson, and Tropp (SIREV 2011) contains extensive analysis, and has made it a very popular method. The typical complexity for a rank-$r$ approximation of $m imes n$ matrices is $O(mn\log n+(m+n)r^2)$ for dense matrices. The classical Nystr{ö}m method is much faster, but applicable only to positive semidefinite matrices. This work studies a generalization of Nystr{ö}m method applicable to general matrices, and shows that (i) it has near-optimal approximation quality comparable to competing methods, (ii) the computational cost is the near-optimal $O(mn\log n+r^3)$ for dense matrices, with small hidden constants, and (iii) crucially, it can be implemented in a numerically stable fashion despite the presence of an ill-conditioned pseudoinverse. Numerical experiments illustrate that generalized Nystr{ö}m can significantly outperform state-of-the-art methods, especially when $r\gg 1$, achieving up to a 10-fold speedup. The method is also well suited to updating and downdating the matrix.

Motivation & Objective

• To generalize the Nyström method, originally limited to positive semidefinite matrices, to general matrices while preserving its speed and accuracy advantages.
• To eliminate the $O(mr^2)$ orthogonalization cost inherent in randomized SVD methods like HMT and Tropp, especially beneficial when $r$ is large.
• To ensure numerical stability despite the use of a pseudoinverse in the approximation formula, which is prone to ill-conditioning.
• To enable efficient matrix updating and downdating, crucial for streaming and incremental learning applications.
• To achieve near-optimal approximation accuracy comparable to HMT and Tropp, but with significantly reduced computational cost.

Proposed method

• Proposes a generalized Nyström approximation: $\hat{A}_r = A X (X^T A X)^\dagger X^T$, where $X$ is a random sketch matrix, generalizing the classical Nyström method to non-PSD matrices.
• Uses a random sketch matrix $X \in \mathbb{R}^{n \times r}$ to compute $A X$ and $X^T A X$, avoiding full orthogonalization of $A X$.
• Employs a randomized range finder to compute $Q = \text{orth}(A X)$, then uses $Q$ to form the approximation $A Q (Q^T A Q)^\dagger Q^T$, which is mathematically equivalent to the generalized Nyström formula.
• Leverages structured random matrices (e.g., SRHT, SRFT, subsampled DCT) to accelerate matrix-vector products and achieve $O(mn\log n)$ sampling cost.
• Introduces a stability analysis showing that roundoff errors have negligible impact on the approximation, ensuring numerical robustness despite the pseudoinverse.
• Enables efficient matrix updating by reusing existing $X$ and $A X$ computations, avoiding recomputation from scratch.

Experimental results

Research questions

• RQ1 Can the Nyström method’s speed, accuracy, and single-pass nature be generalized to non-positive semidefinite matrices?
• RQ2 Is it possible to achieve near-optimal approximation quality with $O(mn\log n + r^3)$ complexity and minimal orthogonalization cost for general matrices?
• RQ3 Can a Nyström-like method be numerically stable despite the use of a potentially ill-conditioned pseudoinverse?
• RQ4 How does the performance of the generalized Nyström method scale with increasing rank $r$, especially in comparison to HMT and Tropp?
• RQ5 Can the method support efficient matrix updating and downdating in streaming or incremental settings?

Key findings

• Generalized Nyström achieves up to a 10-fold speedup over HMT and Tropp for large $r$, with the speedup increasing as $r$ grows.
• The method maintains approximation accuracy comparable to HMT and Tropp, with the relative error in Frobenius norm nearly identical across methods.
• Numerical experiments confirm that roundoff errors have negligible impact on the generalized Nyström approximation, validating its numerical stability.
• For a $50,000 \times 50,000$ dense matrix with geometrically decaying singular values, generalized Nyström significantly outperforms HMT and Tropp in runtime, especially at high ranks.
• In sparse matrix experiments, generalized Nyström still achieves substantial speedup, though the method does not enforce sparsity in the factors, suggesting potential for integration with sparse sketching.
• Matrix updating experiments show that generalized Nyström enables efficient incremental updates by reusing existing sketches and products, avoiding full recomputation.