[Paper Review] Toward a unified theory of sparse dimensionality reduction in Euclidean space

Jean Bourgain, Sjoerd Dirksen+1 more|arXiv (Cornell University)|Nov 11, 2013

Sparse and Compressive Sensing Techniques72 references6 citations

TL;DR

This paper develops a unified theoretical framework for sparse dimensionality reduction in Euclidean space using sparse Johnson-Lindenstrauss transforms (SJLT), showing that a new geometric complexity parameter governs the required sparsity $ s $ and target dimension $ m $ to preserve norms of vectors in a set $ T $ up to $ 1+\varepsilon $. The key contribution is a sparse analog of Gordon's theorem that unifies results in compressed sensing, subspace embeddings, and manifold learning with tight, data-dependent bounds.

ABSTRACT

Let $Φ\in\mathbb{R}^{m imes n}$ be a sparse Johnson-Lindenstrauss transform [KN14] with $s$ non-zeroes per column. For a subset $T$ of the unit sphere, $\varepsilon\in(0,1/2)$ given, we study settings for $m,s$ required to ensure $$ \mathop{\mathbb{E}}_Φ\sup_{x\in T} \left|\|Φx\|_2^2 - 1 ight| < \varepsilon , $$ i.e. so that $Φ$ preserves the norm of every $x\in T$ simultaneously and multiplicatively up to $1+\varepsilon$. We introduce a new complexity parameter, which depends on the geometry of $T$, and show that it suffices to choose $s$ and $m$ such that this parameter is small. Our result is a sparse analog of Gordon's theorem, which was concerned with a dense $Φ$ having i.i.d. Gaussian entries. We qualitatively unify several results related to the Johnson-Lindenstrauss lemma, subspace embeddings, and Fourier-based restricted isometries. Our work also implies new results in using the sparse Johnson-Lindenstrauss transform in numerical linear algebra, classical and model-based compressed sensing, manifold learning, and constrained least squares problems such as the Lasso.

Motivation & Objective

• To develop a general, data-adaptive theory for sparse dimensionality reduction that improves upon worst-case bounds in the Johnson-Lindenstrauss lemma.
• To identify a new geometric complexity parameter that captures the intrinsic structure of a set $ T $ on the unit sphere, enabling tighter bounds on $ m $ and $ s $.
• To unify disparate results in compressed sensing, subspace embeddings, and manifold learning under a single theoretical framework for sparse transforms.
• To provide a sparse analog of Gordon’s theorem, which previously applied only to dense Gaussian matrices.
• To enable improved bounds in applications such as constrained least squares, manifold learning, and numerical linear algebra by leveraging data geometry.

Proposed method

• Introduce a new complexity parameter based on the geometry of the set $ T $, defined via entropy numbers and dual Sudakov minoration.
• Use chaining arguments and tail bounds on the error of the sparse transform to control the supremum of $ |\|\Phi x\|_2^2 - 1| $ over $ x \in T $.
• Apply duality and majorizing measures theory to avoid logarithmic losses in bounds, improving over prior techniques.
• Establish that if the new complexity parameter is small, then $ m $ and $ s $ can be chosen such that $ \mathbb{E}_\Phi \sup_{x\in T} |\|\Phi x\|_2^2 - 1| < \varepsilon $.
• Leverage the structure of the SJLT as a randomly signed adjacency matrix of a biregular bipartite graph to derive probabilistic concentration bounds.
• Prove that norm preservation on tangent vectors of a manifold implies geodesic distance preservation, using bi-Lipschitz properties of the map.

Experimental results

Research questions

• RQ1 What geometric complexity parameter governs the required sparsity $ s $ and target dimension $ m $ for a sparse Johnson-Lindenstrauss transform to preserve norms of all vectors in a set $ T $?
• RQ2 Can a sparse analog of Gordon’s theorem be established for sparse transforms with i.i.d. entries, replacing dense Gaussian matrices?
• RQ3 How can existing results in compressed sensing, subspace embeddings, and manifold learning be qualitatively unified under a single theoretical framework?
• RQ4 What are the quantitative limitations of current bounds, and where can logarithmic losses be avoided through improved use of majorizing measures or entropy duality?
• RQ5 Can alternative graph structures, such as biregular graphs, yield better bounds on $ s $ and $ m $ for specific data sets $ T $?

Key findings

• The required number of non-zero entries per column $ s $ and target dimension $ m $ depend on a new geometric complexity parameter of the set $ T $, which captures its intrinsic structure.
• For $ k $-sparse vectors, the paper shows that $ m \gtrsim k(\log n)^2 $ and $ s \gtrsim (\log n)^2 $ suffice for $ \varepsilon $-norm preservation with high probability.
• For a $ d $-dimensional linear subspace, the required $ m \gtrsim d(\log n)^2 $, and $ s \gtrsim (\log n)^2 $, matching known bounds up to logarithmic factors.
• For a manifold $ \mathcal{M} $, if $ \Phi $ preserves the norm of all unit tangent vectors up to $ 1\pm\varepsilon $, then it preserves geodesic distances up to $ 1+\varepsilon $ with high probability.
• The paper identifies that current bounds lose logarithmic factors due to chaining and duality of entropy numbers, and suggests that improved techniques could yield linear $ s \sim \varepsilon^{-1} $ dependence instead of quadratic.
• The use of majorizing measures theory could avoid logarithmic losses in the bound on the $ \gamma_2 $ functional, potentially improving the dependence on $ \varepsilon $ and $ n $.