[Paper Review] Optimal approximate matrix product in terms of stable rank

Michael B. Cohen, Jelani Nelson+1 more|arXiv (Cornell University)|Jul 8, 2015

Sparse and Compressive Sensing TechniquesEngineering29 references64 citations

TL;DR

This paper establishes that spectral error approximate matrix multiplication (AMM) can be achieved using a random, data-oblivious dimensionality-reducing map $Π$ with $m = O(\tilde{r}/\varepsilon^2)$ rows, where $\tilde{r}$ is the stable rank (squared ratio of Frobenius to operator norm) of the input matrices. The key contribution is showing that any oblivious subspace embedding (OSE) distribution that satisfies the moment method condition for dimension $2\tilde{r}$ suffices for AMM, generalizing prior results beyond subgaussian sketches to include fast and sparse embeddings.

ABSTRACT

We prove, using the subspace embedding guarantee in a black box way, that one can achieve the spectral norm guarantee for approximate matrix multiplication with a dimensionality-reducing map having $m = O( ilde{r}/\varepsilon^2)$ rows. Here $ ilde{r}$ is the maximum stable rank, i.e. squared ratio of Frobenius and operator norms, of the two matrices being multiplied. This is a quantitative improvement over previous work of [MZ11, KVZ14], and is also optimal for any oblivious dimensionality-reducing map. Furthermore, due to the black box reliance on the subspace embedding property in our proofs, our theorem can be applied to a much more general class of sketching matrices than what was known before, in addition to achieving better bounds. For example, one can apply our theorem to efficient subspace embeddings such as the Subsampled Randomized Hadamard Transform or sparse subspace embeddings, or even with subspace embedding constructions that may be developed in the future. Our main theorem, via connections with spectral error matrix multiplication shown in prior work, implies quantitative improvements for approximate least squares regression and low rank approximation. Our main result has also already been applied to improve dimensionality reduction guarantees for $k$-means clustering [CEMMP14], and implies new results for nonparametric regression [YPW15]. We also separately point out that the proof of the "BSS" deterministic row-sampling result of [BSS12] can be modified to show that for any matrices $A, B$ of stable rank at most $ ilde{r}$, one can achieve the spectral norm guarantee for approximate matrix multiplication of $A^T B$ by deterministically sampling $O( ilde{r}/\varepsilon^2)$ rows that can be found in polynomial time. The original result of [BSS12] was for rank instead of stable rank. Our observation leads to a stronger version of a main theorem of [KMST10].

Motivation & Objective

• To resolve the open question of whether stable rank, rather than rank, governs the optimal dimensionality reduction for approximate matrix multiplication with spectral error.
• To characterize the class of random and deterministic dimensionality-reducing maps $\Pi$ that ensure spectral norm error in $A^T B$ approximation.
• To unify and generalize prior results on sketching matrices (e.g., subgaussian, sparse, fast Johnson-Lindenstrauss) by replacing rank-based bounds with stable rank-based bounds.
• To demonstrate that existing analyses of sketching methods—especially those relying on moment bounds—automatically yield stable rank guarantees for AMM.
• To show that deterministic row-sampling matrices with $O(\tilde{r}/\varepsilon^2)$ non-zero entries can achieve spectral norm AMM, improving upon prior rank-based guarantees.

Proposed method

• Introduces a characterization linking the moment method analysis of oblivious subspace embeddings (OSEs) to their applicability in approximate matrix multiplication (AMM).
• Shows that if an OSE distribution $\mathcal{D}$ satisfies a $(\varepsilon, \delta, 2\tilde{r})$-OSE condition via moment bounds, then it guarantees spectral error AMM for matrices of stable rank $\tilde{r}$.
• Applies this characterization to known sketching matrices (e.g., subgaussian, sparse, fast Hadamard), showing their stable rank-based bounds are optimal and generalizable.
• Develops a deterministic algorithm based on the BSS row-sampling framework, modified to use stable rank instead of rank, yielding $O(\tilde{r}/\varepsilon^2)$ non-zero entries.
• Uses barrier functions and matrix perturbation theory (Sherman-Morrison) to analyze the convergence of a randomized rounding process for constructing the sketching matrix.
• Leverages connections to prior work on spectral error matrix multiplication to derive improved bounds for low-rank approximation, regression, and kernel methods.

Experimental results

Research questions

• RQ1 Can the dimensionality reduction for approximate matrix multiplication be characterized in terms of stable rank rather than rank?
• RQ2 Does every oblivious subspace embedding (OSE) that satisfies moment bounds for dimension $k$ also guarantee AMM for matrices of stable rank $k$?
• RQ3 Can the analysis of fast and sparse sketching matrices (e.g., Subsampled Randomized Hadamard, sparse JL) be generalized from rank to stable rank?
• RQ4 Is there a deterministic construction of a sketching matrix with $O(\tilde{r}/\varepsilon^2)$ non-zero entries that achieves spectral norm AMM?
• RQ5 Can the BSS deterministic row-sampling result be strengthened from rank to stable rank, and what are the implications for low-rank approximation and regression?

Key findings

• A random, data-oblivious sketching matrix $\Pi$ with $m = O(\tilde{r}/\varepsilon^2)$ rows suffices for spectral error approximate matrix multiplication, where $\tilde{r}$ is the stable rank of the input matrices.
• Any oblivious subspace embedding (OSE) distribution that satisfies the moment method condition for dimension $2\tilde{r}$ guarantees AMM for matrices of stable rank $\tilde{r}$, generalizing prior results beyond subgaussian sketches.
• The result applies to a broad class of sketching matrices, including sparse, fast, and structured ones (e.g., Subsampled Randomized Hadamard), enabling stable rank-based bounds in their analyses.
• A deterministic sketching matrix with $O(\tilde{r}/\varepsilon^2)$ non-zero entries can be computed in polynomial time and achieves spectral norm AMM, improving upon the original BSS result which used rank instead of stable rank.
• The main result implies faster low-rank approximation for popular kernels in machine learning, such as Gaussian and Sobolev kernels, and improves dimensionality reduction for $k$-means clustering.
• The framework also yields improved bounds for approximate least squares regression and nonparametric regression when combined with existing results.