[Paper Review] Dimensionality Reduction for k-Means Clustering and Low Rank Approximation

Michael B. Cohen, Sam Elder+3 more|arXiv (Cornell University)|Oct 24, 2014

Sparse and Compressive Sensing Techniques41 references14 citations

TL;DR

This paper introduces projection-cost preserving sketches that enable dimensionality reduction for k-means clustering and low-rank approximation with (1+ε) relative error. By reducing data to O(k) dimensions via random projections, SVD, or column sampling, it accelerates any k-means or low-rank algorithm while preserving approximation quality, achieving a (9+ε) approximation for k-means in O(log k / ε²) dimensions—first such result independent of input size and sublinear in k.

ABSTRACT

We show how to approximate a data matrix $\mathbf{A}$ with a much smaller sketch $\mathbf{ ilde A}$ that can be used to solve a general class of constrained k-rank approximation problems to within $(1+ε)$ error. Importantly, this class of problems includes $k$-means clustering and unconstrained low rank approximation (i.e. principal component analysis). By reducing data points to just $O(k)$ dimensions, our methods generically accelerate any exact, approximate, or heuristic algorithm for these ubiquitous problems. For $k$-means dimensionality reduction, we provide $(1+ε)$ relative error results for many common sketching techniques, including random row projection, column selection, and approximate SVD. For approximate principal component analysis, we give a simple alternative to known algorithms that has applications in the streaming setting. Additionally, we extend recent work on column-based matrix reconstruction, giving column subsets that not only `cover' a good subspace for $\bv{A}$, but can be used directly to compute this subspace. Finally, for $k$-means clustering, we show how to achieve a $(9+ε)$ approximation by Johnson-Lindenstrauss projecting data points to just $O(\log k/ε^2)$ dimensions. This gives the first result that leverages the specific structure of $k$-means to achieve dimension independent of input size and sublinear in $k$.

Motivation & Objective

• To develop dimensionality reduction techniques that preserve the cost of k-rank approximations for matrices, enabling faster computation of k-means clustering and low-rank approximation.
• To provide a general framework—projection-cost preserving sketches—that ensures any k-dimensional subspace approximation on the sketch is nearly equivalent to that on the original matrix.
• To achieve (1+ε) relative error for k-means and low-rank problems using only O(k) dimensions, significantly reducing dimensionality compared to prior work.
• To enable efficient distributed and streaming algorithms by minimizing communication and storage through dimensionality reduction.

Proposed method

• Introduces the concept of a projection-cost preserving sketch, where the cost of projecting onto any k-dimensional subspace is multiplicatively preserved up to (1+ε) error.
• Uses random projections, column sampling, and approximate SVD to construct such sketches efficiently, with provable guarantees.
• Applies Johnson-Lindenstrauss-type projections to reduce data to O(log k / ε²) dimensions while maintaining (9+ε) approximation for k-means clustering.
• Leverages existing randomized SVD and sampling algorithms, showing that approximate SVDs and inexact sampling probabilities can be used within the framework.
• Designs a distributed algorithm that communicates only Õ(s log d log k) bits by first reducing to Õ(k/ε²) dimensions and then to O(k/ε) dimensions via non-oblivious projection.
• Extends column-based matrix reconstruction by selecting column subsets that not only span a good subspace but also allow direct computation of the low-rank approximation.

Experimental results

Research questions

• RQ1 Can dimensionality reduction be used to achieve (1+ε) relative error for k-means clustering while reducing data to O(k) dimensions?
• RQ2 Is it possible to construct projection-cost preserving sketches that preserve the cost of k-rank approximations with only O(k) columns, even when the original matrix has high rank?
• RQ3 Can Johnson-Lindenstrauss projections achieve a (9+ε) approximation for k-means clustering in O(log k / ε²) dimensions, independent of input size and sublinear in k?
• RQ4 Can column sampling techniques be used to refine coarse approximate SVDs into relative-error low-rank approximations?
• RQ5 Can the proposed sketching framework be applied to distributed and streaming settings with minimal communication?

Key findings

• The paper achieves a (9+ε) approximation for k-means clustering using Johnson-Lindenstrauss projection to O(log k / ε²) dimensions, the first such result with dimension independent of input size and sublinear in k.
• It provides (1+ε) relative error for k-means and low-rank approximation using sketches of size O(k/ε), via random projection, column sampling, or approximate SVD.
• The proposed projection-cost preserving sketches ensure that the cost of any k-dimensional subspace approximation on the sketch is within (1+ε) of the original matrix.
• A distributed k-means algorithm is designed that communicates only Õ(s log d log k) bits by first reducing to Õ(k/ε²) dimensions and then to O(k/ε) dimensions.
• The framework enables efficient streaming and distributed computation by allowing independent processing of data partitions and minimal communication of sketch matrices.
• The paper extends column-based matrix reconstruction by showing that selected columns can both span a good subspace and be used directly to compute the low-rank approximation.