[Paper Review] Semi-Supervised Generation with Cluster-aware Generative Models

Lars Maaløe, M. Fraccaro+1 more|arXiv (Cornell University)|Apr 3, 2017

Generative Adversarial Networks and Image Synthesis25 references12 citations

TL;DR

This paper proposes the Cluster-aware Generative Model (CaGeM), a variational autoencoder that leverages unlabelled data to learn structured, cluster-aware representations in deep latent spaces, and uses limited labelled data to refine these clusters. The model achieves state-of-the-art log-likelihood performance of −79.38 nats on permutation-invariant MNIST and improves generative modeling by disentangling factors of variation through explicit clustering in higher layers.

ABSTRACT

Deep generative models trained with large amounts of unlabelled data have proven to be powerful within the domain of unsupervised learning. Many real life data sets contain a small amount of labelled data points, that are typically disregarded when training generative models. We propose the Cluster-aware Generative Model, that uses unlabelled information to infer a latent representation that models the natural clustering of the data, and additional labelled data points to refine this clustering. The generative performances of the model significantly improve when labelled information is exploited, obtaining a log-likelihood of -79.38 nats on permutation invariant MNIST, while also achieving competitive semi-supervised classification accuracies. The model can also be trained fully unsupervised, and still improve the log-likelihood performance with respect to related methods.

Motivation & Objective

• To improve deep generative modeling by incorporating limited labelled data into unsupervised VAEs without overfitting.
• To address the challenge of underutilized latent layers in deep VAEs by promoting activation of higher stochastic layers through structured clustering.
• To enable semi-supervised generation where labelled data enhances generative performance beyond classification accuracy.
• To model natural data clustering in deep representations using a discrete latent variable, improving disentanglement and generative quality.

Proposed method

• Extends a standard VAE with a discrete latent variable in the top layer to model natural clustering in data representations.
• Uses unlabelled data to infer cluster structure via the discrete variable, forming a cluster-aware prior.
• Incorporates labelled data to refine cluster assignments, improving generative modeling through class-conditional clustering.
• Applies variational inference with reparameterization gradients to optimize the ELBO, enabling end-to-end training.
• Employs a hierarchical inference network with bottom-up dependencies to approximate the posterior distribution.
• Uses a two-stage training scheme that first learns from unlabelled data and then fine-tunes with labelled data to avoid overfitting.

Experimental results

Research questions

• RQ1 Can limited labelled data improve the generative performance of a deep generative model beyond classification accuracy?
• RQ2 How can unlabelled data be used to structure deep latent representations into meaningful clusters?
• RQ3 Can a discrete latent variable in the top layer of a VAE improve the activation and utility of deeper stochastic layers?
• RQ4 Does cluster-aware representation learning lead to better log-likelihood and disentanglement in generative models?
• RQ5 Can a single model achieve competitive performance in both semi-supervised generation and semi-supervised classification?

Key findings

• CaGeM achieves a log-likelihood of −79.38 nats on permutation-invariant MNIST, setting a new state of the art for semi-supervised generative models.
• The model improves log-likelihood performance over comparable VAE-based models on the OMNIGLOT dataset by leveraging both unlabelled and labelled data.
• Even when trained fully unsupervised, CaGeM outperforms baseline VAEs in log-likelihood, demonstrating improved representation learning.
• The inclusion of a discrete latent variable in the top layer significantly reduces the pruning effect on deeper stochastic layers, enabling better feature abstraction.
• The model achieves competitive semi-supervised classification accuracy, showing dual utility in generation and classification tasks.
• Labelled data refines the cluster structure in the latent space, leading to more disentangled and meaningful representations.