[Paper Review] Sampling from large matrices: an approach through geometric functional analysis

Mark Rudelson, Roman Vershynin|arXiv (Cornell University)|Mar 21, 2005

Sparse and Compressive Sensing TechniquesEngineering21 citations

TL;DR

This paper presents a sampling-based method to approximate large matrices using random submatrices, achieving optimal sample complexity of $ O(r\log r) $, where $ r $ is the numerical rank. It proves that a random submatrix of this size can approximate the original matrix in spectral norm with high probability, enabling efficient low-rank approximation and SVD computation with near-linear time and space complexity.

ABSTRACT

We study random submatrices of a large matrix A. We show how to approximately compute A from its random submatrix of the smallest possible size O(r log r) with a small error in the spectral norm, where r = ||A||_F^2 / ||A||_2^2 is the numerical rank of A. The numerical rank is always bounded by, and is a stable relaxation of, the rank of A. This yields an asymptotically optimal guarantee in an algorithm for computing low-rank approximations of A. We also prove asymptotically optimal estimates on the spectral norm and the cut-norm of random submatrices of A. The result for the cut-norm yields a slight improvement on the best known sample complexity for an approximation algorithm for MAX-2CSP problems. We use methods of Probability in Banach spaces, in particular the law of large numbers for operator-valued random variables.

Motivation & Objective

• To develop a method for approximating large matrices using only a small random submatrix, minimizing sample complexity.
• To establish asymptotically optimal bounds on the spectral and cut norms of random submatrices.
• To enable one-pass or two-pass algorithms for low-rank approximation and SVD on matrices stored externally, with minimal memory and time usage.
• To improve sample complexity for MAX-2CSP approximation algorithms via cut-norm estimates.
• To provide theoretical guarantees using tools from geometric functional analysis and probability in Banach spaces.

Proposed method

• Sample $ d = O\left(\frac{r}{\varepsilon^4\delta}\log\frac{r}{\varepsilon^4\delta}\right) $ rows from matrix $ A $, with replacement and probabilities proportional to squared $ \ell^2 $-norms of rows.
• Use the singular value decomposition (SVD) of the sampled submatrix $ \tilde{A} $ to compute the projection $ P_k $ onto its top $ k $ left singular vectors.
• Approximate the original matrix $ A $ by $ AP_k $, which yields a low-rank approximation in the spectral norm.
• Apply the law of large numbers for operator-valued random variables to control the spectral norm of the approximation error.
• Use concentration inequalities and symmetrization techniques (e.g., Rademacher and Gaussian processes) to bound the expected operator norm of random submatrices.
• Leverage geometric functional analysis tools, including the use of $ \ell^2 $-norms and $ \ell^1 $-norms of singular values, to derive bounds on the numerical rank $ r = \|A\|_F^2 / \|A\|_2^2 $.

Experimental results

Research questions

• RQ1 What is the minimal sample size required to approximate a large matrix $ A $ in the spectral norm using a random submatrix?
• RQ2 Can the spectral norm error of a low-rank approximation based on a random submatrix be bounded with high probability?
• RQ3 How does the numerical rank $ r = \|A\|_F^2 / \|A\|_2^2 $ relate to the optimal sample complexity for matrix approximation?
• RQ4 What are the asymptotically optimal bounds on the spectral and cut norms of random submatrices?
• RQ5 Can the cut-norm estimate improve the sample complexity for MAX-2CSP approximation algorithms?

Key findings

• The paper establishes that $ O(r\log r) $ sampling is sufficient to approximate $ A $ in the spectral norm with high probability, where $ r $ is the numerical rank.
• The approximation error satisfies $ \|A - AP_k\|_2 \leq \sigma_{k+1}(A) + \varepsilon\|A\|_2 $ with high probability, where $ \varepsilon $ controls the additive error.
• The method enables one-pass or two-pass algorithms for low-rank approximation and SVD with $ O(n + m) $ memory and time complexity, polynomial in $ r $ and $ k $.
• The spectral norm of the random submatrix is bounded via symmetrization and concentration, yielding $ \mathbb{E}\|\tilde{A}\|_2 \leq C\|A\|_2\sqrt{\log r} $ under appropriate sampling.
• The cut-norm of the random submatrix is bounded using similar techniques, leading to a slight improvement in sample complexity for MAX-2CSP approximation.
• The analysis shows that the expected spectral norm of the error operator is controlled by $ \sqrt{\log q} \cdot \|A\|_{(n/q)} + \sqrt{q/n} $, where $ q $ is the number of sampled rows.