[Paper Review] CUR Low Rank Approximation at Deterministic Sublinear Cost

Qi Luan, Victor Y. Pan+1 more|arXiv (Cornell University)|Jul 21, 2019

Matrix Theory and Algorithms8 references4 citations

TL;DR

This paper presents deterministic sublinear-cost algorithms for CUR low-rank approximation (LRA) of symmetric positive semidefinite (SPSD) matrices, achieving relative error within a factor of $(1+\epsilon)n$ of the optimal bound. The method leverages maximal volume submatrices via a single loop of Cross-Approximation (C-A) iterations, providing the first formal support for the empirical efficiency of sublinear LRA in this class of matrices.

ABSTRACT

A matrix algorithm runs at {\em sublinear cost} if it uses much fewer memory cells and arithmetic operations than the input matrix has entries. Such algorithms are indispensable for Big Data Mining and Analysis. Quite typically in that area the input matrices are so immense that realistically one can only access a small fraction of all their entries but can access and process at sublinear cost their Low Rank Approximation {\em (LRA)}. Can, however, we compute LRA at sublinear cost? Adversary argument shows that the output of any algorithm running at sublinear cost is extremely far from LRA of the worst case input matrices and even of the matrices of small families of our Appendix, but we prove that some deterministic sublinear cost algorithms output reasonably close LRA in a memory efficient form of CUR LRA if an input matrix admits LRA and is Symmetric Positive Semidefinite or is very close to a low rank matrix. The latter result is technically simple but provides some (very limited but long overdue) support for the well-known empirical efficiency of sublinear cost LRA by means of Cross-Approximation. We demonstrate the power of application of such LRA by turning the Fast Multipole celebrated Method into Superfast Multipole Method. The design and analysis of our algorithms rely on extensive prior study of the link of LRA of a matrix to maximization of its volume.

Motivation & Objective

• Address the long-standing gap in theoretical justification for the empirical success of sublinear-cost LRA algorithms in Big Data applications.
• Provide formal guarantees for CUR low-rank approximation when the input matrix is symmetric positive semidefinite (SPSD) or close to low rank.
• Demonstrate that deterministic sublinear-cost algorithms can produce reasonably accurate LRA for SPSD matrices, overcoming the adversary argument that rules out such guarantees for general matrices.
• Establish a theoretical link between maximal volume submatrices and effective CUR LRA, supporting the practical use of Cross-Approximation.
• Enable the transformation of the Fast Multipole Method into a Superfast Multipole Method via efficient LRA, showcasing real-world impact.

Proposed method

• Use a single loop of Cross-Approximation (C-A) iterations to compute a $p \times r$ or $r \times q$ submatrix with maximal volume, ensuring near-optimal CUR LRA for rank-$r$ matrices.
• Leverage the connection between maximal volume submatrices and CUR LRA, where the volume of a submatrix determines the quality of the approximation.
• Design deterministic algorithms that run in sublinear cost by accessing only $O((m+n)r)$ entries, significantly fewer than the $mn$ entries of the full matrix.
• Apply the theoretical framework to transform the Fast Multipole Method into a Superfast Multipole Method, reducing computational complexity.
• Prove that for SPSD matrices admitting LRA, the relative error of the output CUR LRA is within a factor of $(1+\epsilon)n$ of the optimal bound, provided $n/r \in O(1/\epsilon)$.
• Use volume maximization as a proxy for low-rank approximation quality, avoiding reliance on randomized sampling or iterative refinement.

Experimental results

Research questions

• RQ1 Can deterministic sublinear-cost algorithms achieve reasonably accurate CUR low-rank approximation for symmetric positive semidefinite matrices?
• RQ2 What is the theoretical justification for the empirical success of Cross-Approximation (C-A) in computing LRA at sublinear cost?
• RQ3 How does the maximal volume of a submatrix relate to the quality of the resulting CUR LRA in the context of sublinear algorithms?
• RQ4 Can sublinear-cost LRA be formally guaranteed for any class of matrices, despite the adversary argument ruling out such guarantees for general matrices?
• RQ5 To what extent can the Fast Multipole Method be accelerated using sublinear-cost LRA, and what are the implications for numerical methods?

Key findings

• The proposed deterministic sublinear-cost algorithms achieve a relative error in CUR LRA that is within a factor of $(1+\epsilon)n$ of the optimal bound for any $n \times n$ symmetric positive semidefinite matrix admitting LRA.
• The bound holds under the condition that $n/r \in O(1/\epsilon)$, ensuring the approximation quality is controllable for a given tolerance $\epsilon$.
• A single loop of C-A iterations computes a submatrix with maximal volume within a bounded factor, enabling a CUR LRA with acceptable error for matrices close to low rank.
• This result provides the first formal theoretical support for the long-observed empirical efficiency of sublinear-cost LRA via Cross-Approximation.
• The method enables the transformation of the Fast Multipole Method into a Superfast Multipole Method, demonstrating practical impact in numerical linear algebra.
• The approach avoids randomized sampling and instead relies on volume maximization, offering a deterministic and memory-efficient alternative to existing randomized sublinear LRA algorithms.