[Paper Review] A Sparse Johnson--Lindenstrauss Transform

Anirban Dasgupta, Ravi Kumar+1 more|arXiv (Cornell University)|Apr 23, 2010

Sparse and Compressive Sensing Techniques19 references19 citations

TL;DR

This paper introduces a sparse Johnson-Lindenstrauss transform using a hash-based construction that achieves O(1/ε) non-zero entries per column—significantly sparser than the classical O(1/ε²)—enabling ˜O(1/ε) update time per non-zero element. The method leverages hashing with local densification and proves a matching lower bound, offering faster dimensionality reduction for sparse vectors and streaming workloads.

ABSTRACT

Dimension reduction is a key algorithmic tool with many applications including nearest-neighbor search, compressed sensing and linear algebra in the streaming model. In this work we obtain a {\em sparse} version of the fundamental tool in dimension reduction --- the Johnson--Lindenstrauss transform. Using hashing and local densification, we construct a sparse projection matrix with just $ ilde{O}(\frac{1}ε)$ non-zero entries per column. We also show a matching lower bound on the sparsity for a large class of projection matrices. Our bounds are somewhat surprising, given the known lower bounds of $Ω(\frac{1}{ε^2})$ both on the number of rows of any projection matrix and on the sparsity of projection matrices generated by natural constructions. Using this, we achieve an $ ilde{O}(\frac{1}ε)$ update time per non-zero element for a $(1\pmε)$-approximate projection, thereby substantially outperforming the $ ilde{O}(\frac{1}{ε^2})$ update time required by prior approaches. A variant of our method offers the same guarantees for sparse vectors, yet its $ ilde{O}(d)$ worst case running time matches the best approach of Ailon and Liberty.

Motivation & Objective

• To design a sparse random projection matrix that preserves pairwise distances with high probability while minimizing non-zero entries per column.
• To overcome the known Ω(1/ε²) lower bound on sparsity for i.i.d. random matrices by introducing structured dependencies via hashing.
• To achieve faster update times for sparse vectors and streaming data, particularly in the turnstile model where global transforms are expensive.
• To establish a matching lower bound on sparsity for a broad class of projection matrices, proving the construction is asymptotically optimal.

Proposed method

• Constructs a projection matrix using a hash function that maps each dimension to exactly one hash bucket, introducing controlled dependencies among entries.
• Uses ±1 random variables instead of Gaussians to enable tighter concentration bounds and simplify variance analysis.
• Applies the FKG inequality to handle correlations between error contributions from different hash buckets, ensuring concentration of the total error.
• Employs a replication technique to densify sparse input vectors, reducing the required ∥x∥∞ for the JL property from O(ε) to O(√ε).
• Uses block-Hadamard preconditioning to achieve ˜O(min(nnz(x)/ε, d)) worst-case running time across all vectors.
• Derandomizes the hash function using Nisan’s pseudorandom generator, though this increases update time to ˜O(k) unless more efficient derandomization is found.

Experimental results

Research questions

• RQ1 Can a Johnson-Lindenstrauss transform be constructed with o(1/ε²) non-zero entries per column while preserving (1±ε)-distortion?
• RQ2 Does a hash-based construction with fixed column sparsity and structured dependencies enable better sparsity and faster update times than i.i.d. random matrices?
• RQ3 What is the optimal lower bound on the number of non-zero entries per column for a matrix to satisfy the JL property under general conditions?
• RQ4 Can the update time for sparse vectors in the turnstile streaming model be reduced below ˜O(d) using a sparse, structured projection?
• RQ5 Is the proposed construction optimal, and can a matching lower bound be proven for a broad class of projection matrices?

Key findings

• The paper constructs a sparse Johnson-Lindenstrauss transform with O(1/ε log²(k/δ) log(1/δ)) non-zero entries per column, achieving o(1/ε²) sparsity.
• The method achieves ˜O(1/ε) update time per non-zero input element, significantly improving over the ˜O(1/ε²) time of prior approaches.
• A matching lower bound of Ω(min(1/ε², √log k(d)/ε)) non-zero entries per column is proven for a large class of projection matrices.
• For sparse vectors, the running time is ˜O(nnz(x)/ε), which is faster than existing methods in terms of sparsity ratio and ε-dependence.
• The construction enables ˜O(min(nnz(x)/ε, d)) worst-case time for all vectors, matching the best known bounds for sparse vectors.
• The method is optimal in terms of sparsity and update time, with the lower bound holding even under mild moment conditions on the non-zero entries.