[Paper Review] Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering

Zhuxi Jiang, Yin Zheng+3 more|arXiv (Cornell University)|Nov 16, 2016

Generative Adversarial Networks and Image Synthesis43 references87 citations

TL;DR

This paper proposes Variational Deep Embedding (VaDE), a novel unsupervised generative clustering framework that combines Variational Autoencoders (VAE) with a Gaussian Mixture Model (GMM) prior to jointly learn deep representations and cluster assignments. By modeling data generation through cluster selection, latent embedding sampling, and DNN-based decoding, VaDE optimizes the evidence lower bound (ELBO) via stochastic gradient variational Bayes with reparameterization, achieving state-of-the-art clustering performance and generating realistic samples per cluster without supervision.

ABSTRACT

Clustering is among the most fundamental tasks in computer vision and machine learning. In this paper, we propose Variational Deep Embedding (VaDE), a novel unsupervised generative clustering approach within the framework of Variational Auto-Encoder (VAE). Specifically, VaDE models the data generative procedure with a Gaussian Mixture Model (GMM) and a deep neural network (DNN): 1) the GMM picks a cluster; 2) from which a latent embedding is generated; 3) then the DNN decodes the latent embedding into observables. Inference in VaDE is done in a variational way: a different DNN is used to encode observables to latent embeddings, so that the evidence lower bound (ELBO) can be optimized using Stochastic Gradient Variational Bayes (SGVB) estimator and the reparameterization trick. Quantitative comparisons with strong baselines are included in this paper, and experimental results show that VaDE significantly outperforms the state-of-the-art clustering methods on 4 benchmarks from various modalities. Moreover, by VaDE's generative nature, we show its capability of generating highly realistic samples for any specified cluster, without using supervised information during training. Lastly, VaDE is a flexible and extensible framework for unsupervised generative clustering, more general mixture models than GMM can be easily plugged in.

Motivation & Objective

• To develop a clustering method that jointly learns deep representations and cluster assignments in an unsupervised manner.
• To endow clustering models with generative capability to produce realistic samples conditioned on cluster identities.
• To overcome limitations of existing deep clustering models like DEC, which lack generative modeling and sample synthesis ability.
• To improve clustering performance by modeling the data generation process using a mixture of Gaussians as a prior over latent codes.
• To demonstrate that the proposed framework generalizes across diverse data modalities and remains robust under varying cluster numbers.

Proposed method

• VaDE models data generation as a three-step process: (1) a cluster is selected from a GMM prior, (2) a latent embedding is sampled from the cluster-specific Gaussian distribution, and (3) a deep neural network decodes the latent embedding into observable data.
• The model is trained by maximizing the evidence lower bound (ELBO) of the log-likelihood using the Stochastic Gradient Variational Bayes (SGVB) estimator.
• A separate encoder network is used to map observed data to latent embeddings, enabling end-to-end training via the reparameterization trick.
• The GMM prior replaces the standard single Gaussian prior in VAE, allowing the model to naturally support multiple clusters and model cluster-specific data distributions.
• The optimization process alternates between updating the encoder and decoder networks to improve both representation learning and clustering performance.
• The framework supports flexible prior design, allowing extension to other mixture models beyond GMMs in future work.

Experimental results

Research questions

• RQ1 Can a deep generative model be designed to perform unsupervised clustering while retaining the ability to generate realistic samples?
• RQ2 How does combining VAE with a GMM prior improve clustering performance compared to standard VAE or autoencoder-based clustering?
• RQ3 To what extent can the learned latent representations preserve the intrinsic structure of the data, especially in comparison to non-generative models like DEC?
• RQ4 How does the model perform when the number of clusters is not equal to the true number of classes?
• RQ5 Can the model generate high-quality, diverse samples conditioned on a specific cluster label without any supervised signal?

Key findings

• VaDE significantly outperforms state-of-the-art clustering methods on five diverse benchmarks across different data modalities, demonstrating superior clustering accuracy.
• The model achieves better clustering results than VAE and DEC, with fewer incorrectly clustered samples, especially in the interior regions of clusters.
• Visualizations using t-SNE show that VaDE learns more structured and separable representations than VAE and DEC, with misclassified samples primarily located at cluster boundaries.
• When the number of clusters is set below the true number of classes (e.g., 7 on MNIST), VaDE groups visually similar digits (e.g., 9 and 4, 3 and 8) into the same cluster.
• When the number of clusters exceeds the number of classes (e.g., 14 on MNIST), VaDE discovers sub-classes such as thin vs. thick 0s and upright vs. slanted 1s.
• VaDE generates highly realistic, diverse, and class-conditioned samples, with performance comparable to InfoGAN, despite being trained in a purely unsupervised manner.