[Paper Review] Unsupervised Deep Embedding for Clustering Analysis

Junyuan Xie, Ross Girshick+1 more|arXiv (Cornell University)|Nov 19, 2015

Image Retrieval and Classification Techniques36 references116 citations

TL;DR

This paper proposes Deep Embedded Clustering (DEC), a method that jointly learns deep feature representations and cluster assignments via iterative optimization of a clustering objective using a self-supervised target distribution. DEC achieves state-of-the-art clustering accuracy and robustness on image and text datasets by end-to-end training with backpropagation, outperforming existing unsupervised methods in both performance and scalability.

ABSTRACT

Clustering is central to many data-driven application domains and has been studied extensively in terms of distance functions and grouping algorithms. Relatively little work has focused on learning representations for clustering. In this paper, we propose Deep Embedded Clustering (DEC), a method that simultaneously learns feature representations and cluster assignments using deep neural networks. DEC learns a mapping from the data space to a lower-dimensional feature space in which it iteratively optimizes a clustering objective. Our experimental evaluations on image and text corpora show significant improvement over state-of-the-art methods.

Motivation & Objective

• To address the limitation of traditional clustering methods that rely on handcrafted or linear feature spaces by learning data-driven, non-linear representations for clustering.
• To jointly optimize deep neural network feature learning and cluster assignment in an end-to-end manner without requiring labeled cluster assignments.
• To improve clustering performance and robustness to hyperparameter choices in unsupervised settings where cross-validation is not feasible.
• To scale clustering to large datasets by achieving linear complexity in the number of data points, unlike spectral clustering methods with quadratic or higher complexity.

Proposed method

• DEC uses a deep autoencoder to learn a non-linear mapping from input data to a lower-dimensional embedding space.
• It introduces a soft cluster assignment via a Student's t-distribution-based target distribution that is updated iteratively to refine cluster assignments.
• The model optimizes a Kullback-Leibler (KL) divergence objective between the predicted cluster distribution and the target distribution using stochastic gradient descent.
• The network is trained in an iterative fashion: first initializing with autoencoder features, then refining cluster assignments and updating the embedding via backpropagation.
• The target distribution is dynamically updated based on current soft assignments to encourage confident cluster predictions and improve embedding quality.
• The method is scalable and linear in the number of data points, enabling efficient training on large datasets.

Experimental results

Research questions

• RQ1 Can deep neural networks be trained end-to-end to jointly learn feature representations and cluster assignments in an unsupervised manner?
• RQ2 How does iterative refinement using a self-supervised target distribution improve clustering performance compared to standard clustering methods?
• RQ3 To what extent does DEC reduce sensitivity to hyperparameter choices in the absence of labeled data for validation?
• RQ4 Can the proposed method scale efficiently to large datasets while maintaining high clustering accuracy?

Key findings

• DEC achieves state-of-the-art clustering accuracy on MNIST, STL-10, and REUTERS datasets, significantly outperforming standard and state-of-the-art clustering methods.
• The method demonstrates robustness to hyperparameter settings, with performance consistently improving across different configurations without requiring extensive cross-validation.
• DEC shows a sharp drop in generalizability from 9 to 10 clusters on MNIST, indicating that 9 is the optimal number of clusters, which aligns with the highest NMI score.
• The t-SNE visualization of embedded representations shows increasingly well-separated clusters over training epochs, confirming the model's ability to learn meaningful cluster structures.
• DEC is significantly less sensitive to class imbalance than baseline methods, maintaining high accuracy even when cluster sizes vary widely.
• The ablation study confirms that autoencoder initialization combined with KL divergence minimization leads to substantial performance gains, outperforming k-means and other baselines on autoencoded features.