[Paper Review] Semi-Supervised Learning with Deep Generative Models

Diederik P. Kingma, Danilo Jimenez Rezende+2 more|UvA-DARE (University of Amsterdam)|Jun 20, 2014

Generative Adversarial Networks and Image Synthesis25 references1,521 citations

TL;DR

This paper proposes a deep generative model for semi-supervised learning that leverages variational inference to jointly optimize model and inference parameters, enabling scalable and accurate classification using only a small number of labeled examples. The method achieves state-of-the-art performance on benchmark datasets like SVHN and MNIST by modeling data density and disentangling class-conditional content from style variability.

ABSTRACT

The ever-increasing size of modern data sets combined with the difficulty of obtaining label information has made semi-supervised learning one of the problems of significant practical importance in modern data analysis. We revisit the approach to semi-supervised learning with generative models and develop new models that allow for effective generalisation from small labelled data sets to large unlabelled ones. Generative approaches have thus far been either inflexible, inefficient or non-scalable. We show that deep generative models and approximate Bayesian inference exploiting recent advances in variational methods can be used to provide significant improvements, making generative approaches highly competitive for semi-supervised learning.

Motivation & Objective

• To address the challenge of semi-supervised learning where labeled data is scarce but unlabeled data is abundant.
• To develop a scalable, probabilistic framework that combines deep generative models with approximate Bayesian inference for improved generalization.
• To overcome limitations of prior generative approaches, which were inflexible, inefficient, or non-scalable.
• To enable effective use of data density information to improve decision boundaries beyond supervised learning alone.
• To demonstrate state-of-the-art performance on standard benchmarks using a unified generative model with joint optimization of model and variational parameters.

Proposed method

• The model uses a deep generative architecture with latent variables z to represent data, where x is generated from z and class label y.
• A variational inference framework is employed to approximate the true posterior p(z|x,y), using a recognition model (inference network) to estimate q(z|x,y).
• Stochastic variational inference is applied to jointly optimize both model parameters θ and variational parameters φ, enabling scalability to large datasets.
• The model is trained using a variational lower bound (ELBO) that incorporates both labeled and unlabeled data, maximizing the log-likelihood of observed data under the model.
• The framework supports both inductive and transductive learning by modeling the joint density p(x,y,z) and using the posterior over z for prediction.
• The method enables analogical reasoning by manipulating latent codes z across different classes, demonstrating disentangled representation learning.

Experimental results

Research questions

• RQ1 Can deep generative models with scalable approximate Bayesian inference achieve state-of-the-art performance in semi-supervised learning?
• RQ2 How effectively can generative models leverage unlabeled data to improve classification beyond supervised baselines?
• RQ3 Can variational inference be effectively applied to joint optimization of model and inference parameters in deep generative models for semi-supervised learning?
• RQ4 To what extent do the learned representations disentangle class-conditional content from intra-class variability (e.g., style)?
• RQ5 How does the model’s performance scale with increasing amounts of unlabeled data and varying numbers of labeled examples?

Key findings

• On the SVHN dataset with only 1,000 labeled examples, the proposed method achieved a test accuracy of 77.93% using KNN on generated features, outperforming standard KNN and TSVM baselines.
• The model achieved a test accuracy of 78.71% on the NORB dataset with 1,000 labeled examples using KNN on features from the generative model, significantly outperforming the baseline KNN and TSVM.
• The method demonstrated state-of-the-art performance on benchmark semi-supervised learning tasks, showing that generative models can be competitive with discriminative approaches when combined with deep architectures and variational inference.
• Qualitative analysis showed that the model learns disentangled representations, where latent variables z capture style variations while class labels y control content, enabling analogical image generation across classes.
• The stochastic variational inference algorithm enabled efficient joint optimization of model and variational parameters, making the approach scalable to large datasets.
• The model’s ability to generate analogies—producing images by transferring style from one class to another—demonstrates its capacity for meaningful representation learning.