[Paper Review] Tutorial on Variational Autoencoders

Carl Doersch|arXiv (Cornell University)|Jun 19, 2016

Generative Adversarial Networks and Image Synthesis25 references1,372 citations

TL;DR

This tutorial introduces variational autoencoders (VAEs), derives their objective via variational Bayes, explains the reparameterization trick for training, and extends to conditional VAEs for one-to-many mappings, with MNIST demonstrations.

ABSTRACT

In just three years, Variational Autoencoders (VAEs) have emerged as one of the most popular approaches to unsupervised learning of complicated distributions. VAEs are appealing because they are built on top of standard function approximators (neural networks), and can be trained with stochastic gradient descent. VAEs have already shown promise in generating many kinds of complicated data, including handwritten digits, faces, house numbers, CIFAR images, physical models of scenes, segmentation, and predicting the future from static images. This tutorial introduces the intuitions behind VAEs, explains the mathematics behind them, and describes some empirical behavior. No prior knowledge of variational Bayesian methods is assumed.

Motivation & Objective

• Motivate generative modeling and unsupervised learning of complex distributions.
• Explain how VAEs use latent variables to capture data structure without hand-crafted features.
• Present a tractable training objective that combines reconstruction and regularization.
• Introduce the reparameterization trick to enable gradient-based optimization.

Proposed method

• Define P(X) as the integral over latent z and introduce P(X|z) with a Gaussian likelihood.
• Introduce Q(z|X) to approximate the intractable P(z|X) and derive the evidence lower bound (ELBO).
• Show how the ELBO decomposes into a reconstruction term and a KL-divergence regularizer.
• Use the reparameterization trick to backpropagate through stochastic sampling: z = μ(X) + Σ^{1/2}(X) ε where ε ~ N(0,I).
• Adopt Gaussian Q(z|X) with diagonal Σ and compute the KL term in closed form.
• Extend to conditional VAEs (CVAE) by conditioning the generative process on input X to model P(Y|X).
• Demonstrate testing-time sampling by sampling z ~ N(0,I) and decoding without the encoder.

Experimental results

Research questions

• RQ1 How can we train a generative model with latent variables without requiring explicit posterior calculations?
• RQ2 How can we compute and optimize a tractable objective that approximates the data likelihood?
• RQ3 How does the reparameterization trick enable gradient-based learning in VAEs?
• RQ4 How can VAEs be extended to handle conditional, one-to-many mappings (CVAE)?

Key findings

• VAEs maximize a tractable lower bound on the data likelihood, balancing reconstruction quality and latent regularization.
• The reparameterization trick enables backpropagation through stochastic latent variables, making training via SGD feasible.
• A Gaussian approximate posterior Q(z|X) with a closed-form KL to P(z) yields a computable objective.
• The framework naturally connects to minimum description length and information-theoretic interpretations.
• CVAEs extend VAEs to conditional generation, enabling multi-modal outputs given input X.
• Testing-time sampling from z ~ N(0,I) and decoding produces new data without using the encoder.