[Paper Review] Fast approximation of matrix coherence and statistical leverage

Petros Drineas, Malik Magdon‐Ismail+2 more|arXiv (Cornell University)|Sep 18, 2011

Sparse and Compressive Sensing TechniquesEngineering48 references171 citations

TL;DR

This paper presents a randomized algorithm that computes relative-error approximations to all $ n $ statistical leverage scores of an $ n \times d $ matrix in $ O(nd\log n / \epsilon^2) $ time, significantly faster than the $ O(nd^2) $ time of the standard SVD-based approach. The method leverages Johnson-Lindenstrauss-type random projections and linear sketching to efficiently estimate leverage scores without computing the full SVD, enabling scalable computation in large-scale data analysis and randomized matrix algorithms.

ABSTRACT

The statistical leverage scores of a matrix $A$ are the squared row-norms of the matrix containing its (top) left singular vectors and the coherence is the largest leverage score. These quantities are of interest in recently-popular problems such as matrix completion and Nyström-based low-rank matrix approximation as well as in large-scale statistical data analysis applications more generally; moreover, they are of interest since they define the key structural nonuniformity that must be dealt with in developing fast randomized matrix algorithms. Our main result is a randomized algorithm that takes as input an arbitrary $n imes d$ matrix $A$, with $n \gg d$, and that returns as output relative-error approximations to all $n$ of the statistical leverage scores. The proposed algorithm runs (under assumptions on the precise values of $n$ and $d$) in $O(n d \log n)$ time, as opposed to the $O(nd^2)$ time required by the naïve algorithm that involves computing an orthogonal basis for the range of $A$. Our analysis may be viewed in terms of computing a relative-error approximation to an underconstrained least-squares approximation problem, or, relatedly, it may be viewed as an application of Johnson-Lindenstrauss type ideas. Several practically-important extensions of our basic result are also described, including the approximation of so-called cross-leverage scores, the extension of these ideas to matrices with $n \approx d$, and the extension to streaming environments.

Motivation & Objective

• To develop a fast, scalable algorithm for approximating statistical leverage scores in large matrices, which are critical for matrix completion and low-rank approximation.
• To reduce the computational cost of computing leverage scores from $ O(nd^2) $ to $ O(nd\log n / \epsilon^2) $, enabling practical use in large-scale data analysis.
• To extend the algorithm to streaming and distributed environments, supporting real-time or space-efficient computation.
• To provide relative-error approximations to coherence and cross-leverage scores, which are key structural properties in randomized matrix algorithms.
• To enable efficient sampling of matrix rows proportional to their leverage scores for applications such as matrix sketching and regression.

Proposed method

• The algorithm uses a random projection matrix $ \Pi $ and a linear sketching matrix $ T $ to compress the input matrix $ A $, preserving row norms related to leverage scores.
• It computes $ TA $ and $ \Pi A $ in a single pass over the data, then applies a transformation using $ R^{-1} $ to recover approximate leverage scores.
• The method relies on Johnson-Lindenstrauss-type embeddings to preserve $ \ell_2 $-norms of the rows of the matrix of left singular vectors $ U $, up to a $ (1\pm\epsilon) $ factor.
• Leverage scores are estimated via $ \ell_2 $-sampling techniques, where row sampling probabilities are proportional to $ \|U_{(i)}\|_2^2 $, enabling efficient identification of high-leverage rows.
• For streaming applications, the algorithm uses sketching matrices that support $ F_p $-norm estimation and entropy approximation of the leverage score distribution.
• The algorithm can be extended to estimate coherence (maximum leverage score), cross-leverage scores, and entropy of the leverage score distribution with small space and a single pass.

Experimental results

Research questions

• RQ1 Can statistical leverage scores be approximated in $ o(nd^2) $ time while maintaining relative error guarantees?
• RQ2 How can Johnson-Lindenstrauss-type random projections be used to estimate leverage scores without computing the full SVD?
• RQ3 What is the minimal space and pass complexity required to approximate leverage scores in a streaming setting?
• RQ4 Can the algorithm be extended to estimate coherence, cross-leverage scores, and entropy of the leverage score distribution?
• RQ5 How can efficient row sampling proportional to leverage scores be achieved using linear sketches?

Key findings

• The proposed algorithm computes relative-error approximations to all $ n $ leverage scores in $ O(nd\log n / \epsilon^2) $ time, a significant improvement over the $ O(nd^2) $ time of the standard SVD-based method.
• The algorithm achieves relative-error approximation to the coherence of the matrix in the same time complexity, as coherence is the maximum leverage score.
• The method supports single-pass, streaming computation using $ O(sd\epsilon^{-2}\log^4 n) $ bits of space for $ s $ leverage-score-proportional row samples.
• Entropy of the leverage score distribution can be approximated within additive $ \epsilon $ using $ O(d\epsilon^{-4}\log^6 n\log^{14}(1/\epsilon)) $ bits of space.
• The algorithm enables efficient identification of high-leverage rows (those with $ \|U_{(i)}\|_2^2 \geq \tau\|U\|_F^2 $) using $ O(d\tau^{-1}\epsilon^{-2}\log^3 n\log \tau^{-1}) $ rows in the sketch.
• The approach is robust to rank-deficient matrices and extends naturally to cases where $ n \approx d $, though numerical stability considerations are left for future work.