[Paper Review] Deep Continuous Clustering

Sohil Shah, Vladlen Koltun|arXiv (Cornell University)|Mar 5, 2018

Advanced Clustering Algorithms Research33 references55 citations

TL;DR

Deep Continuous Clustering (DCC) jointly learns nonlinear dimensionality reduction and clustering by training a deep autoencoder under a robust continuous objective that does not require pre-specifying the number of clusters, and shows strong performance on high-dimensional datasets.

ABSTRACT

Clustering high-dimensional datasets is hard because interpoint distances become less informative in high-dimensional spaces. We present a clustering algorithm that performs nonlinear dimensionality reduction and clustering jointly. The data is embedded into a lower-dimensional space by a deep autoencoder. The autoencoder is optimized as part of the clustering process. The resulting network produces clustered data. The presented approach does not rely on prior knowledge of the number of ground-truth clusters. Joint nonlinear dimensionality reduction and clustering are formulated as optimization of a global continuous objective. We thus avoid discrete reconfigurations of the objective that characterize prior clustering algorithms. Experiments on datasets from multiple domains demonstrate that the presented algorithm outperforms state-of-the-art clustering schemes, including recent methods that use deep networks.

Motivation & Objective

• Motivate clustering in high-dimensional spaces where inter-point distances are less informative.
• Develop a joint embedding and clustering framework that optimizes a single continuous objective.
• Avoid requiring a priori knowledge of the number of clusters through a robust continuous clustering formulation.
• Enable scalable optimization via gradient-based solvers and deep networks.

Proposed method

• Embed data into a low-dimensional space with a deep autoencoder parameterized by Θ.
• Constrain the embedding with a reconstruction objective ensuring X can be reconstructed from its low-dimensional representation Y.
• Integrate a robust continuous clustering objective (RCC-based) that uses a set of representatives Z and a pairwise robust loss to drive clustering without discrete reassignments.
• Define a joint objective L(Ω, Z) combining reconstruction loss, data loss, and pairwise clustering loss, optimized via stochastic gradient methods.
• Construct a mutual kNN graph on X and use a nonconvex robust M-estimator to encourage intra-cluster cohesion and inter-cluster separation.
• Initialize using stacked denoising autoencoders and employ continuation to ease optimization, with termination based on stability of cluster structure.

Experimental results

Research questions

• RQ1 Can joint nonlinear embedding and clustering be optimized with a single continuous objective without pre-specifying the number of clusters?
• RQ2 Does integrating deep dimensionality reduction with clustering yield better clustering accuracy than sequential (embedding-then-clustering) approaches?
• RQ3 How does a robust continuous clustering objective perform on high-dimensional data compared to traditional clustering losses?
• RQ4 What is the impact of the latent space dimensionality on clustering performance and robustness across datasets?

Key findings

• DCC achieves on-par or better clustering performance than state-of-the-art baselines across multiple high-dimensional datasets.
• Joint optimization of embedding and clustering yields superior results compared to stagewise SDAE embeddings followed by clustering.
• DCC demonstrates robustness to the dimensionality of the latent space, outperforming some baselines as d increases.
• The approach does not require prior knowledge of the true number of clusters, yet yields well-defined cluster structure in the learned space.
• Convolutional and fully-connected autoencoder configurations show competitive runtime, with conv-DCC faster than several deep clustering alternatives.