[Paper Review] Fast Monte-Carlo Low Rank Approximations for Matrices

Shmuel Friedland, M. Kaveh+2 more|ArXiv.org|Oct 26, 2005

Scientific Research and Discoveries4 references4 citations

TL;DR

This paper proposes a Monte Carlo algorithm for fast, iterative low-rank approximation of large matrices using random sampling of rows or columns. By iteratively updating the approximation with $O(k)$ additional rows/columns, the method guarantees monotonic improvement in Frobenius norm and achieves $O(kmn)$ complexity, significantly outperforming standard SVD on large matrices while converging to near-optimal low-rank approximations in few iterations.

ABSTRACT

In many applications, it is of interest to approximate data, given by mxn matrix A, by a matrix B of at most rank k, which is much smaller than m and n. The best approximation is given by singular value decomposition, which is too time consuming for very large m and n. We present here a Monte Carlo algorithm for iteratively computing a k-rank approximation to the data consisting of mxn matrix A. Each iteration involves the reading of O(k) of columns or rows of A. The complexity of our algorithm is O(kmn). Our algorithm, distinguished from other known algorithms, guarantees that each iteration is a better k-rank approximation than the previous iteration. We believe that this algorithm will have many applications in data mining, data storage and data analysis.

Motivation & Objective

• To address the computational infeasibility of full SVD for very large $m \times n$ matrices in data analysis and data mining applications.
• To develop an iterative algorithm that improves low-rank approximation with each update, ensuring monotonic convergence in approximation quality.
• To reduce the computational cost of $k$-rank approximation from $O(mn \min(m,n))$ to $O(kmn)$ while maintaining high accuracy.
• To enable practical computation of low-rank approximations on matrices too large for deterministic SVD, especially in high-dimensional data settings.

Proposed method

• The algorithm iteratively samples $l = O(k)$ rows or columns from the input matrix $A$ to update the current $k$-rank approximation $B$.
• At each iteration, the method computes a spectral decomposition of a $(k+l) \times (k+l)$ symmetric matrix to refine the approximation.
• The update process ensures that the Frobenius norm $\|B\|_F$ increases monotonically, guaranteeing improvement at every step.
• The algorithm leverages matrix multiplication of $k$-dimensional vectors with $A^T$ or $A$, which can be parallelized for performance gains.
• Sampling strategies include uniform sampling with and without replacement, and weighted sampling based on row gradients in image data.
• The algorithm uses the increasing $\|B\|_F$ as a stopping criterion, indicating convergence toward the optimal $k$-rank approximation.

Experimental results

Research questions

• RQ1 Can a randomized iterative algorithm guarantee monotonic improvement in low-rank approximation quality for large matrices?
• RQ2 How can the computational complexity of $k$-rank approximation be reduced below $O(mn \min(m,n))$ while maintaining high accuracy?
• RQ3 What sampling strategy (with/without replacement, weighted) yields the fastest convergence to the optimal low-rank approximation?
• RQ4 Can the algorithm scale to matrices too large for standard SVD, such as $8000 \times 200$?
• RQ5 To what extent does the iterative update framework outperform non-iterative randomized SVD methods in terms of speed and accuracy?

Key findings

• The algorithm converges to the optimal $k$-rank approximation with relative error approaching the theoretical minimum, as shown on synthetic and real image data.
• For a $3000 \times 500$ synthetic matrix with rank 500, the relative error converged to within 1% of the optimal error in about 5 iterations.
• On the Cameraman and Liftingbody images, the algorithm achieved relative error ratios of 1.083 and 1.08 respectively with only a fraction of the data sampled.
• For a large $8000 \times 200$ random matrix with $k=100$, the algorithm achieved a 42x speedup over deterministic SVD while maintaining relative error within 10% of the optimal.
• The algorithm achieved speedups of 1.145, 8, 3.33, and 42 on four different data sets, with relative error ratios consistently below 1.1.
• The use of sampling without replacement led to faster convergence than with replacement, confirming theoretical expectations.