[Paper Review] Sparsity Lower Bounds for Dimensionality Reducing Maps

Jelani Nelson, Huy L. Nguyên|arXiv (Cornell University)|Nov 5, 2012

Sparse and Compressive Sensing Techniques40 references4 citations

TL;DR

This paper establishes near-optimal lower bounds on the sparsity required for dimensionality-reducing linear maps, including Johnson-Lindenstrauss transforms, restricted isometry property (RIP) matrices, and oblivious subspace embeddings. It proves that sparse JL matrices require Ω(ε⁻¹ log n / log(1/ε)) nonzeros per column when m = o(n / log(1/ε)), and that RIP matrices with optimal row count m = O(k log(n/k)) must have Ω(k log(n/k)) nonzeros per column, showing sparsity is fundamentally limited in such constructions.

ABSTRACT

We give near-tight lower bounds for the sparsity required in several dimensionality reducing linear maps. First, consider the JL lemma which states that for any set of n vectors in R there is a matrix A in R^{m x d} with m = O(eps^{-2}log n) such that mapping by A preserves pairwise Euclidean distances of these n vectors up to a 1 +/- eps factor. We show that there exists a set of n vectors such that any such matrix A with at most s non-zero entries per column must have s = Omega(eps^{-1}log n/log(1/eps)) as long as m < O(n/log(1/eps)). This bound improves the lower bound of Omega(min{eps^{-2}, eps^{-1}sqrt{log_m d}}) by [Dasgupta-Kumar-Sarlos, STOC 2010], which only held against the stronger property of distributional JL, and only against a certain restricted class of distributions. Meanwhile our lower bound is against the JL lemma itself, with no restrictions. Our lower bound matches the sparse Johnson-Lindenstrauss upper bound of [Kane-Nelson, SODA 2012] up to an O(log(1/eps)) factor. Next, we show that any m x n matrix with the k-restricted isometry property (RIP) with constant distortion must have at least Omega(klog(n/k)) non-zeroes per column if the number of the rows is the optimal value m = O(klog (n/k)), and if k < n/polylog n. This improves the previous lower bound of Omega(min{k, n/m}) by [Chandar, 2010] and shows that for virtually all k it is impossible to have a sparse RIP matrix with an optimal number of rows. Lastly, we show that any oblivious distribution over subspace embedding matrices with 1 non-zero per column and preserving all distances in a d dimensional-subspace up to a constant factor with constant probability must have at least Omega(d^2) rows. This matches one of the upper bounds in [Nelson-Nguyen, 2012] and shows the impossibility of obtaining the best of both of constructions in that work, namely 1 non-zero per column and Õ(d) rows.

Motivation & Objective

• To understand the fundamental limits of sparsity in dimensionality-reducing linear maps for large-scale data processing.
• To close the gap between known upper bounds and lower bounds on sparsity for sparse Johnson-Lindenstrauss transforms.
• To show that optimal-row RIP matrices cannot be sparse under general conditions.
• To resolve the tradeoff between sparsity and number of rows in oblivious subspace embeddings.
• To provide tight bounds that rule out constructions combining 1 non-zero per column and Õ(d) rows for subspace embeddings.

Proposed method

• Uses a probabilistic construction of a hard instance of n vectors in Rd to prove lower bounds on column sparsity for Johnson-Lindenstrauss matrices.
• Applies an argument based on net-packing and covering numbers to derive lower bounds for RIP matrices with optimal row count.
• Employs a second moment method and variance analysis to bound the probability that a sparse matrix preserves ℓ₂ norms of vectors in a d-dimensional subspace.
• Leverages the Johnson-Lindenstrauss lemma and its known upper bounds to derive matching lower bounds up to logarithmic factors.
• Analyzes OSNAP-type distributions—matrices with i.i.d. entries in {0, ±1/√s}—to establish tradeoffs between sparsity and number of rows.
• Uses the γ₂ functional and Gaussian width to characterize the complexity of vector sets and derive bounds on required matrix dimensions.

Experimental results

Research questions

• RQ1 What is the minimum number of non-zero entries per column required in a Johnson-Lindenstrauss matrix to preserve pairwise distances with distortion 1±ε, when the number of rows is suboptimal?
• RQ2 Can restricted isometry property (RIP) matrices with the optimal number of rows m = O(k log(n/k)) achieve constant column sparsity for k-sparse vectors?
• RQ3 Is it possible to construct an oblivious subspace embedding with only one non-zero per column and Õ(d) rows that preserves all ℓ₂ norms in a d-dimensional subspace?
• RQ4 What is the tradeoff between the number of rows and column sparsity in OSNAP-type distributions for embedding sets of vectors with given geometric complexity?
• RQ5 Can the sparsity of dimensionality-reducing maps be improved beyond black-box applications of the Johnson-Lindenstrauss lemma?

Key findings

• For Johnson-Lindenstrauss transforms, any matrix preserving pairwise distances with distortion 1±ε must have at least Ω(ε⁻¹ log n / log(1/ε)) nonzeros per column if m = o(n / log(1/ε)).
• This lower bound matches the sparse JL upper bound of Kane and Nelson (SODA 2012) up to an O(log(1/ε)) factor, establishing near-optimality.
• For RIP matrices with optimal row count m = O(k log(n/k)), the column sparsity must be Ω(k log(n/k)) when k < n / polylog n, showing sparsity is impossible in this regime.
• The lower bound for RIP matrices improves upon Chandar (2010) from Ω(min{k, n/m}) to Ω(k log(n/k)), showing that sparse RIP matrices with optimal rows cannot exist for most k.
• For oblivious subspace embeddings with one non-zero per column, at least Ω(d²) rows are required to preserve distances in a d-dimensional subspace, ruling out constructions with Õ(d) rows and unit column sparsity.
• The paper shows that combining 1 non-zero per column and Õ(d) rows is impossible for subspace embeddings, resolving a key open question from Nelson and Nguyen (2012).