[Paper Review] Stochastic Sparse Subspace Clustering

Ying Chen, Chun-Guang Li+1 more|arXiv (Cornell University)|May 4, 2020

Face and Expression Recognition64 references4 citations

TL;DR

This paper proposes Stochastic Sparse Subspace Clustering (S3COMP), a scalable and flexible method that uses dropout in the self-expressive model to induce denser representations, thereby improving subspace connectivity and reducing over-segmentation in sparse subspace clustering. By reformulating the optimization as a consensus problem over small subproblems, S3COMP achieves state-of-the-art performance on large-scale datasets with improved clustering accuracy and connectivity.

ABSTRACT

State-of-the-art subspace clustering methods are based on self-expressive model, which represents each data point as a linear combination of other data points. By enforcing such representation to be sparse, sparse subspace clustering is guaranteed to produce a subspace-preserving data affinity where two points are connected only if they are from the same subspace. On the other hand, however, data points from the same subspace may not be well-connected, leading to the issue of over-segmentation. We introduce dropout to address the issue of over-segmentation, which is based on randomly dropping out data points in self-expressive model. In particular, we show that dropout is equivalent to adding a squared $\ell_2$ norm regularization on the representation coefficients, therefore induces denser solutions. Then, we reformulate the optimization problem as a consensus problem over a set of small-scale subproblems. This leads to a scalable and flexible sparse subspace clustering approach, termed Stochastic Sparse Subspace Clustering, which can effectively handle large scale datasets. Extensive experiments on synthetic data and real world datasets validate the efficiency and effectiveness of our proposal.

Motivation & Objective

• To address over-segmentation in sparse subspace clustering (SSC), where points from the same subspace may not be well-connected due to overly sparse representations.
• To improve subspace connectivity by introducing dropout as a regularization mechanism that induces denser coefficient solutions.
• To develop a scalable and memory-efficient algorithm capable of handling ultra-large-scale datasets by solving small-scale subproblems through stochastic optimization.
• To provide a flexible framework that maintains subspace-preserving properties while enhancing clustering performance on real-world and synthetic data.

Proposed method

• Dropout is applied by randomly removing columns of the data matrix X during coefficient estimation, which implicitly adds an ℓ₂ regularization to the representation coefficients.
• The method reformulates the sparse subspace clustering problem as a consensus optimization over multiple small-scale subproblems generated via stochastic sampling.
• The final representation is computed as a weighted average of the solutions from multiple stochastic trials, with non-zero entries aggregated based on their frequency and magnitude.
• The approach leverages the equivalence between dropout and ℓ₂ regularization, which promotes denser and more robust coefficient vectors compared to pure ℓ₁-based sparsity.
• The algorithm is implemented iteratively, with convergence monitored via stability in support size and clustering accuracy across outer iterations.
• Parameter tuning is guided by eigenvalue gap analysis and dataset size, with higher dropout rates δ used for larger datasets.

Experimental results

Research questions

• RQ1 Can dropout be effectively used to improve subspace connectivity in sparse subspace clustering without sacrificing sparsity or scalability?
• RQ2 Does the stochastic optimization framework based on small subproblems maintain subspace-preserving properties while reducing over-segmentation?
• RQ3 How does the proposed method compare to SSCOMP and other state-of-the-art methods in terms of clustering accuracy and connectivity on real-world datasets?
• RQ4 What is the impact of hyperparameters δ (dropout rate) and λ (regularization strength) on the performance and stability of the algorithm?

Key findings

• S3COMP and S3COMP-C achieve significantly higher average connectivity (c̄) than SSCOMP on all real-world datasets, with improvements up to 0.1883 on MNIST70000 and 0.1883 on GTSRB.
• On Extended Yale B and COIL100, S3COMP-C achieves a mean connectivity of 0.0667 and 0.0077 respectively, outperforming SSCOMP’s 0.0381 and 0.0060.
• The clustering accuracy of S3COMP-C stabilizes within a few outer iterations, with support size becoming stable after only 2–3 iterations on all datasets.
• The method demonstrates robustness to parameter choices within the range [0.1, 1.0] for λ, with performance less sensitive to λ when connectivity improvements are notable.
• S3COMP-C achieves the highest clustering accuracy on MNIST4000 (95.2%), MNIST10000 (94.8%), and MNIST70000 (94.5%), outperforming SSCOMP and EnSC.
• The use of a consensus solution via frequency-weighted averaging (S3COMP*) further improves connectivity and accuracy compared to simple averaging.