[Paper Review] Sparse Quantized Spectral Clustering

Zhenyu Liao, Romain Couillet+1 more|arXiv (Cornell University)|Oct 3, 2020

Sparse and Compressive Sensing Techniques47 references4 citations

TL;DR

This paper proposes a data-dependent, sparse quantized spectral clustering method that drastically reduces computational and storage costs while preserving clustering performance. Using random matrix theory, it shows that carefully designed entry-wise nonlinear transformations (sparsification, quantization, binarization) maintain the informative eigenspectrum of the kernel matrix, enabling near-optimal clustering even with 90% of entries zeroed out or reduced to 1 bit per entry.

ABSTRACT

Given a large data matrix, sparsifying, quantizing, and/or performing other entry-wise nonlinear operations can have numerous benefits, ranging from speeding up iterative algorithms for core numerical linear algebra problems to providing nonlinear filters to design state-of-the-art neural network models. Here, we exploit tools from random matrix theory to make precise statements about how the eigenspectrum of a matrix changes under such nonlinear transformations. In particular, we show that very little change occurs in the informative eigenstructure even under drastic sparsification/quantization, and consequently that very little downstream performance loss occurs with very aggressively sparsified or quantized spectral clustering. We illustrate how these results depend on the nonlinearity, we characterize a phase transition beyond which spectral clustering becomes possible, and we show when such nonlinear transformations can introduce spurious non-informative eigenvectors.

Motivation & Objective

• To analyze the impact of entry-wise nonlinear transformations—such as sparsification, quantization, and binarization—on the eigenspectrum of kernel matrices in spectral clustering.
• To characterize the conditions under which such transformations preserve the informative eigenstructure necessary for clustering.
• To identify phase transitions in the dominant eigenvalue-eigenvector pair based on signal-to-noise ratio (SNR).
• To optimize quantization and binarization operators for minimal performance loss under fixed storage budgets.
• To demonstrate that selective, data-driven sparsification outperforms uniform random sparsification in both performance and efficiency.

Proposed method

• Applies random matrix theory (RMT) to derive the asymptotic eigenvalue distribution of the transformed kernel matrix K = f(X^T X / √p) / √p as n, p → ∞.
• Uses a mixture model where data points are i.i.d. N(±μ, I_p) to model signal and noise components.
• Introduces a general entry-wise thresholding/quantization operator f to model sparsification, quantization, and binarization.
• Derives theoretical conditions under which the dominant eigenvector retains class-structure information (i.e., is not asymptotically orthogonal to the true class label vector).
• Optimizes the thresholding function f for minimal performance loss under fixed storage constraints (e.g., bits per non-zero entry).
• Validates theoretical findings via simulations on real-world datasets like MNIST, comparing performance across different f functions and sparsity levels.

Experimental results

Research questions

• RQ1 How does sparsification or quantization affect the eigenspectrum of a kernel matrix in spectral clustering, especially in high-dimensional asymptotic regimes?
• RQ2 At what signal-to-noise ratio (SNR) does the dominant eigenvector become informative for clustering, and when does it become purely noisy?
• RQ3 Can data-dependent, non-uniform sparsification outperform uniform random sparsification in terms of clustering accuracy and computational efficiency?
• RQ4 What is the optimal quantization or binarization scheme that minimizes performance loss under a fixed storage budget?
• RQ5 Under what conditions do nonlinear transformations introduce spurious, non-informative eigenvectors?

Key findings

• For a signal-to-noise ratio (SNR) ∥μ∥² exceeding a threshold γ, the dominant eigenvalue becomes isolated and its eigenvector captures class-structure information; otherwise, it is asymptotically orthogonal to the true class labels.
• With optimal thresholding, binary quantization (f3) achieves performance within 1% of the full-precision linear kernel, even at 54% sparsity (ε ≈ 0.54).
• The optimal threshold for binary f3 is sopt ≈ 0.43, independent of ρ or c, and yields ν/a²₁ ≈ 1.24, significantly better than sign(t) (ν/a²₁ ≈ 1.57).
• For M ≥ 5, the optimal quantized f2 achieves ν/a²₁ close to 1, indicating near-theoretical performance with minimal loss.
• On MNIST, binary f3 maintains near-optimal clustering error even when up to 90% of matrix entries are discarded, with computational time reduced by up to 80%.
• Quantized f2 and binary f3 outperform sparse f1 with the same storage size, demonstrating that quantization enables better performance-complexity trade-offs than sparsification alone.