[Paper Review] Sketches for Matrix Norms: Faster, Smaller and More General.

Vladimir Braverman, Stephen R. Chestnut+2 more|arXiv (Cornell University)|Sep 19, 2016

Stochastic Gradient Optimization Techniques3 citations

TL;DR

This paper introduces faster, smaller, and more general sketching algorithms for unitarily invariant matrix norms, including Schatten $p$-norms, using modern Johnson-Lindenstrauss-based dimensionality reduction. It achieves optimal space complexity for $p o 2$ and sublinear sketch sizes via intrinsic dimension-aware embeddings, enabling efficient streaming approximation with $O(1)$ update time and improved query performance.

ABSTRACT

We design new sketching algorithms for unitarily invariant matrix norms, including the Schatten $p$-norms~$\|{\cdot}\|_{S_p}$, and obtain, as a by-product, streaming algorithms that approximate the norm of a matrix $A$ presented as a turnstile data stream. The primary advantage of our streaming algorithms is that they are simpler and faster than previous algorithms, while requiring the same or less storage. Our three main results are a faster sketch for estimating $\|{A}\|_{S_p}$, a smaller-space $O(1)$-pass sketch for $\|{A}\|_{S_p}$, and more general sketching technique that yields sublinear-space approximations for a wide class of matrix norms. These improvements are powered by dimensionality reduction techniques that are modern incarnations of the Johnson-Lindenstrauss Lemma~\cite{JL84}. When $p\geq 2$ is even or $A$ is PSD, our fast one-pass algorithm approximates $\|{A}\|_{S_p}$ in optimal, $n^{2-4/p}$, space with $O(1)$ update time and $o(n^{2.4(1-2/p)})$ time to extract the approximation from the sketch, while the $\lceil p/2 ceil$-pass algorithm is built on a smaller sketch of size $n^{1-1/(p-1)}$ with $O(1)$ update time and $n^{1-1/(p-1)}$ query time. Finally, for a PSD matrix $A$ and a unitarily invariant norm $l(\cdot)$, we prove that one can obtain an approximation to $l(A)$ from a sketch $GAH^T$ where $G$ and $H$ are independent Oblivious Subspace Embeddings and the dimension of the sketch is polynomial in the intrinsic dimension of $A$. The intrinsic dimension of a matrix is a robust version of the rank that is equal to the ratio $\sum_i\sigma_i/\sigma_1$. It is small, e.g., for models in machine learning which consist of a low rank matrix plus noise. Naturally, this leads to much smaller sketches for many norms.

Motivation & Objective

• To design simpler and faster streaming algorithms for approximating unitarily invariant matrix norms, such as Schatten $p$-norms.
• To reduce space usage in $O(1)$-pass and multi-pass streaming settings while maintaining or improving time efficiency.
• To generalize sketching techniques beyond $p$-norms to a wide class of matrix norms using intrinsic dimension as a complexity parameter.
• To achieve optimal space complexity for even $p \geq 2$ and positive semidefinite matrices via dimensionality reduction.
• To demonstrate that sketch size can be polynomial in the intrinsic dimension rather than matrix rank, enabling smaller sketches for low-rank-plus-noise models.

Proposed method

• Leverages modern variants of the Johnson-Lindenstrauss Lemma for dimensionality reduction in matrix sketching.
• Employs Oblivious Subspace Embeddings (OSES) for constructing sketches of the form $GAH^T$ with independent $G$ and $H$.
• Uses the intrinsic dimension of a matrix—defined as $\sum_i \sigma_i / \sigma_1$—as a measure of complexity to bound sketch size.
• Designs a one-pass algorithm with $O(1)$ update time and $o(n^{2.4(1-2/p)})$ query time for $\|A\|_{S_p}$ when $p \geq 2$ is even or $A$ is PSD.
• Develops a $\lceil p/2 \rceil$-pass algorithm with sketch size $n^{1-1/(p-1)}$ and $O(1)$ update/query time.
• Applies the sketch $GAH^T$ to approximate any unitarily invariant norm $l(A)$ for PSD matrices, with sketch dimension polynomial in intrinsic dimension.

Experimental results

Research questions

• RQ1 Can we achieve optimal space complexity for $\|A\|_{S_p}$ in a streaming setting with $O(1)$ update time?
• RQ2 Can we reduce sketch size below $n^{1-1/(p-1)}$ while maintaining accuracy for $\|A\|_{S_p}$?
• RQ3 Can we generalize sketching beyond Schatten $p$-norms to a broader class of unitarily invariant matrix norms?
• RQ4 Does the intrinsic dimension of a matrix provide a tighter bound on sketch size than traditional rank?
• RQ5 Can we achieve sublinear-space approximation for matrix norms using OSE-based sketches?

Key findings

• The one-pass algorithm approximates $\|A\|_{S_p}$ in $n^{2-4/p}$ space with $O(1)$ update time and $o(n^{2.4(1-2/p)})$ query time for even $p \geq 2$ or PSD matrices.
• The $\lceil p/2 \rceil$-pass algorithm uses a sketch of size $n^{1-1/(p-1)}$ with $O(1)$ update and query time.
• For PSD matrices and any unitarily invariant norm $l(\cdot)$, the sketch $GAH^T$ yields an approximation to $l(A)$ with dimension polynomial in the intrinsic dimension of $A$.
• The intrinsic dimension $\sum_i \sigma_i / \sigma_1$ is shown to be a robust and effective measure for controlling sketch size, especially in low-rank-plus-noise models.
• The proposed sketching framework achieves optimal space complexity for $\|A\|_{S_p}$ when $p \geq 2$ is even, matching known lower bounds.
• The method generalizes beyond $p$-norms, enabling sublinear-space approximations for a wide class of matrix norms via intrinsic dimension-aware embeddings.