[Paper Review] Beyond Linear Subspace Clustering: A Comparative Study of Nonlinear Manifold Clustering Algorithms

Maryam Abdolali, Nicolas Gillis|arXiv (Cornell University)|Mar 19, 2021

Face and Expression Recognition186 references43 citations

TL;DR

This paper presents a comprehensive comparative study of nonlinear manifold clustering algorithms, categorizing them into locality preserving, kernel-based, and neural network-based approaches. It evaluates representative methods on synthetic and real-world datasets, revealing that neural network-based methods often underperform without post-processing, while kernel and locality-preserving methods show stronger empirical performance and robustness.

ABSTRACT

Subspace clustering is an important unsupervised clustering approach. It is based on the assumption that the high-dimensional data points are approximately distributed around several low-dimensional linear subspaces. The majority of the prominent subspace clustering algorithms rely on the representation of the data points as linear combinations of other data points, which is known as a self-expressive representation. To overcome the restrictive linearity assumption, numerous nonlinear approaches were proposed to extend successful subspace clustering approaches to data on a union of nonlinear manifolds. In this comparative study, we provide a comprehensive overview of nonlinear subspace clustering approaches proposed in the last decade. We introduce a new taxonomy to classify the state-of-the-art approaches into three categories, namely locality preserving, kernel based, and neural network based. The major representative algorithms within each category are extensively compared on carefully designed synthetic and real-world data sets. The detailed analysis of these approaches unfolds potential research directions and unsolved challenges in this field.

Motivation & Objective

• To address the limitations of linear subspace clustering in modeling complex, nonlinear data structures.
• To provide a systematic taxonomy for classifying nonlinear manifold clustering algorithms into three main categories: locality preserving, kernel-based, and neural network-based.
• To empirically compare the performance of representative algorithms across these categories on both synthetic and real-world datasets.
• To identify key challenges and open problems in nonlinear manifold clustering, including parameter sensitivity, scalability, and lack of theoretical guarantees.
• To guide future research by highlighting unsolved issues such as clustering intersecting manifolds and improving robustness to outliers.

Proposed method

• Proposes a new taxonomy to classify nonlinear subspace clustering methods into three categories: locality preserving, kernel-based, and neural network-based.
• Reviews and compares representative algorithms from each category, including LMC, KSC, and DSC-Net, using self-expressiveness in the original or transformed space.
• Employs spectral clustering on the learned coefficient matrix to perform final clustering, following the standard pipeline of linear SC.
• Uses Laplacian regularization in locality-preserving methods to maintain local structure in the embedding space.
• Applies kernel methods to map data into a higher-dimensional feature space where linear clustering becomes feasible.
• Utilizes deep autoencoders in neural network-based approaches to learn nonlinear representations, with a focus on enforcing self-expressive property in the latent space.

Experimental results

Research questions

• RQ1 How do different classes of nonlinear manifold clustering algorithms—locality preserving, kernel-based, and neural network-based—compare in terms of clustering accuracy and robustness?
• RQ2 To what extent do neural network-based approaches achieve subspace-preserving representations without ad-hoc post-processing?
• RQ3 How sensitive are nonlinear clustering algorithms to hyperparameters such as neighborhood size, kernel bandwidth, or training epochs?
• RQ4 What are the computational bottlenecks in existing nonlinear SC methods, especially regarding the n×n coefficient matrix storage and Gram matrix computation?
• RQ5 Can existing nonlinear SC methods reliably cluster data from intersecting or closely spaced manifolds?

Key findings

• Neural network-based nonlinear SC methods, such as DSC-Net, often fail to produce subspace-preserving representations without a dedicated post-processing step, indicating insufficient inductive bias from self-expressiveness in the latent space.
• Kernel-based and locality-preserving methods generally outperform neural network-based approaches on synthetic data with independent or well-separated manifolds.
• The performance of all methods degrades significantly when manifolds are close or intersecting, especially for locality-preserving and kernel-based methods sensitive to neighborhood and kernel parameters.
• Computational cost remains a major bottleneck, particularly due to the O(n²) complexity of neighborhood graphs and the n×n coefficient matrix storage, limiting scalability beyond 10,000 data points.
• There is a lack of theoretical guarantees for nonlinear SC, especially for neural network-based methods, making model behavior difficult to interpret and prone to trivial embeddings.
• Robustness to gross corruptions and outliers remains a critical challenge, as most methods rely on the Frobenius norm for data fitting, which is sensitive to noise and occlusions.