[Paper Review] Sketching as a Tool for Numerical Linear Algebra

David P. Woodruff|arXiv (Cornell University)|Nov 17, 2014

Matrix Theory and Algorithms119 references585 citations

TL;DR

This paper presents sketching as a powerful technique for accelerating numerical linear algebra algorithms by compressing large matrices into smaller sketches using random projections. It achieves near-optimal performance for least squares regression, robust regression, low-rank approximation, and graph sparsification, with theoretical guarantees on error and runtime, including sublinear and polylogarithmic time complexity for many problems.

ABSTRACT

This survey highlights the recent advances in algorithms for numerical linear algebra that have come from the technique of linear sketching, whereby given a matrix, one first compresses it to a much smaller matrix by multiplying it by a (usually) random matrix with certain properties. Much of the expensive computation can then be performed on the smaller matrix, thereby accelerating the solution for the original problem. In this survey we consider least squares as well as robust regression problems, low rank approximation, and graph sparsification. We also discuss a number of variants of these problems. Finally, we discuss the limitations of sketching methods.

Motivation & Objective

• To develop and analyze sketching-based algorithms that significantly accelerate classical numerical linear algebra problems such as regression and low-rank approximation.
• To provide theoretical bounds on approximation error and runtime for sketching methods in over-constrained settings where n ≫ d.
• To investigate the limits of sketching through communication and streaming lower bounds, especially for Schatten norms and subspace embeddings.
• To identify open problems in robust low-rank approximation, distributed computation, and sketching of Schatten-1 norms.
• To unify and survey recent advances in sketching for numerical linear algebra, emphasizing practical efficiency and theoretical rigor.

Proposed method

• Uses random matrix projections (sketches) to compress input matrices A ∈ ℝ^{n×d} into smaller matrices S A ∈ ℝ^{r×d} with r ≪ n, preserving key structural properties.
• Employs subspace embeddings via matrices with sub-Gaussian or heavy-tailed entries (e.g., Gaussian, Cauchy, Exponential) to preserve ℓ₂ and ℓ₁ norms of A x for all x.
• Applies sketching to the normal equations in least squares, replacing A^T A with (S A)^T (S A) to reduce computation from O(nd²) to O(rd²).
• Introduces adaptive sampling and CUR decomposition to construct low-rank approximations with provable error bounds using sketching for column selection.
• Leverages sketching in distributed and streaming models to reduce communication and space complexity, achieving O(sdk/ε) communication for s-players.
• Uses matrix sketches to approximate Schatten norms, especially the nuclear norm (p=1), and proves lower bounds on sketch dimension for constant-factor approximation.

Experimental results

Research questions

• RQ1 Can sketching be used to achieve (1+ε)-approximate solutions for least squares regression in sublinear time with high probability?
• RQ2 What is the optimal sketching dimension for approximating the Schatten-1 norm of a matrix up to a constant factor?
• RQ3 Can sketching be extended to robust regression with ℓ₁-loss (L1-Regression) with provable error and efficiency guarantees?
• RQ4 What are the communication and space lower bounds for sketching in distributed and streaming settings for low-rank approximation and regression?
• RQ5 Is there a polynomial-time algorithm for robust low-rank approximation under the entry-wise ℓ₁-norm (i.e., ∥A - Ã∥₁ ≤ (1+ε)∥A - Aₖ∥₁)?

Key findings

• Sketching enables (1+ε)-approximate least squares regression in O(nnz(A) + n·poly(k/ε)) time, significantly improving over classical O(nd²) methods.
• For ℓ₁-regression, sketching with Cauchy or Exponential random variables provides subspace embeddings that support efficient sampling-based solvers.
• A polylogarithmic sketch dimension suffices for approximating the Frobenius norm of low-rank matrices, with error bounds matching known optimal results.
• The paper establishes an Ω(n^{1/2}) lower bound for sketching the Schatten-1 norm (nuclear norm) to constant factor, and shows improved bounds under matrix sketch constraints.
• An O(sdk/ε) communication protocol is presented for distributed low-rank approximation, matching the known Ω(sdk) lower bound up to logarithmic factors.
• The paper identifies open problems, including the existence of efficient algorithms for ℓ₁-low-rank approximation and proving an Ω(sdk/ε) communication lower bound for distributed sketching.