[Paper Review] Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders

Nat Dilokthanakul, Pedro A. M. Mediano+5 more|arXiv (Cornell University)|Nov 8, 2016

Generative Adversarial Networks and Image Synthesis23 references498 citations

TL;DR

The paper proposes a Gaussian Mixture Variational Autoencoder (GMVAE) to perform unsupervised clustering within a VAE framework, analyzes over-regularisation, applies a minimum information constraint to mitigate it, and demonstrates competitive unsupervised clustering on synthetic data, MNIST, and SVHN with interpretable latent clusters.

ABSTRACT

We study a variant of the variational autoencoder model (VAE) with a Gaussian mixture as a prior distribution, with the goal of performing unsupervised clustering through deep generative models. We observe that the known problem of over-regularisation that has been shown to arise in regular VAEs also manifests itself in our model and leads to cluster degeneracy. We show that a heuristic called minimum information constraint that has been shown to mitigate this effect in VAEs can also be applied to improve unsupervised clustering performance with our model. Furthermore we analyse the effect of this heuristic and provide an intuition of the various processes with the help of visualizations. Finally, we demonstrate the performance of our model on synthetic data, MNIST and SVHN, showing that the obtained clusters are distinct, interpretable and result in achieving competitive performance on unsupervised clustering to the state-of-the-art results.

Motivation & Objective

• Motivate unsupervised clustering within deep generative models using a multimodal latent prior.
• Propose a Gaussian mixture prior in a VAE and derive a tractable ELBO for inference.
• Identify and mitigate over-regularisation in GMVAE via minimum information constraint.
• Demonstrate that GMVAE learns interpretable, distinct latent clusters on image datasets.

Proposed method

• Define a generative model where latent x is drawn from a Gaussian mixture conditioned on w, with a discrete cluster z selecting a mixture component.
• Use a recognition model to approximate the posterior with a factorised variational family and derive an ELBO that includes a conditional prior term for z.
• Compute the z-posterior using pβ(z|x,w) and estimate the conditional prior term without sampling discrete variables, enabling standard backpropagation.
• Address over-regularisation by applying a Kingma et al. 2016-style modified ELBO that caps the z-prior term with a threshold λ, promoting cluster spread before regularization activates.
• Evaluate with Monte Carlo estimates and show that increasing Monte Carlo samples M and clusters K improves stability and performance.
• Demonstrate generative capabilities by showing that different z components correspond to digits and that w modulates style in MNIST/SVHN.

Experimental results

Research questions

• RQ1 Can a GMVAE with a Gaussian mixture prior achieve unsupervised clustering within a VAE framework?
• RQ2 How does over-regularisation affect GMVAE clustering, and can minimum information constraints mitigate it?
• RQ3 Do GMVAE clusters correspond to meaningful, interpretable factors in image data (MNIST/SVHN) beyond synthetic examples?

Key findings

• GMVAE can learn distinct, interpretable clusters on MNIST and SVHN, achieving competitive unsupervised clustering performance.
• Without mitigation, the z-prior term leads to cluster degeneracy where all data collapse into a single large cluster.
• A Kingma et al.-style modification to the ELBO that caps the z-prior term prevents early cluster merging and improves clustering spread.
• Increasing the number of clusters K and Monte Carlo samples M improves robustness and clustering accuracy.
• GMVAE’s latent components align with digits, where z controls class and w influences style, enabling interpretable generation.
• On MNIST with K=10, GMVAE achieves competitive unsupervised classification accuracy compared to state-of-the-art methods.