[Paper Review] Ladder Variational Autoencoders

Casper Kaae Sønderby, Tapani Raiko+3 more|arXiv (Cornell University)|Feb 6, 2016

Generative Adversarial Networks and Image Synthesis19 references216 citations

TL;DR

The paper proposes the Ladder Variational Autoencoder (LVAE), a deep generative model that improves variational inference by integrating top-down, data-dependent corrections into the inference process, using shared parameters between inference and generative models. This results in significantly tighter likelihood bounds and state-of-the-art log-likelihood performance on MNIST, OMNIGLOT, and NORB, while learning deeper, more distributed hierarchical representations than standard VAEs.

ABSTRACT

Variational Autoencoders are powerful models for unsupervised learning. However deep models with several layers of dependent stochastic variables are difficult to train which limits the improvements obtained using these highly expressive models. We propose a new inference model, the Ladder Variational Autoencoder, that recursively corrects the generative distribution by a data dependent approximate likelihood in a process resembling the recently proposed Ladder Network. We show that this model provides state of the art predictive log-likelihood and tighter log-likelihood lower bound compared to the purely bottom-up inference in layered Variational Autoencoders and other generative models. We provide a detailed analysis of the learned hierarchical latent representation and show that our new inference model is qualitatively different and utilizes a deeper more distributed hierarchy of latent variables. Finally, we observe that batch normalization and deterministic warm-up (gradually turning on the KL-term) are crucial for training variational models with many stochastic layers.

Motivation & Objective

• To improve the training and performance of deep variational autoencoders with multiple stochastic layers.
• To address the limitations of purely bottom-up inference in VAEs, which struggles to utilize deep hierarchical structures.
• To investigate whether structured inference with top-down feedback enhances generative modeling and representation learning.
• To evaluate the impact of batch normalization and deterministic warm-up on training deep stochastic models.

Proposed method

• Proposes a new inference model, the Ladder VAE (LVAE), that uses a top-down, precision-weighted correction of the generative distribution using data-dependent likelihood estimates.
• Introduces a recursive inference mechanism where each layer's posterior is computed by combining bottom-up likelihood with top-down prior, using shared parameters with the generative model.
• Employs a reparameterization trick for stochastic backpropagation through Gaussian latent variables and computes KL divergence analytically when possible.
• Uses a hierarchical prior structure where each latent layer depends on the next, enabling deep, structured representations.
• Applies batch normalization and a deterministic warm-up schedule to stabilize training of deep stochastic models.
• Employs a variational lower bound with importance weighting for tighter likelihood estimation.

Experimental results

Research questions

• RQ1 Can a structured inference model with top-down feedback improve the generative performance of deep VAEs?
• RQ2 How does the LVAE's inference mechanism compare to purely bottom-up VAE inference in terms of log-likelihood and representation quality?
• RQ3 What is the impact of batch normalization and warm-up on training deep VAEs and LVAEs?
• RQ4 Does the LVAE learn deeper and more distributed hierarchical representations than standard VAEs?
• RQ5 Can the LVAE achieve state-of-the-art performance without complex inference architectures like normalizing flows or auxiliary networks?

Key findings

• The LVAE achieves a test set log-likelihood of −85.81 on MNIST, significantly outperforming standard VAEs and matching or surpassing more complex models like IWAE and VAEs with normalizing flows.
• On OMNIGLOT, the LVAE achieves a test log-likelihood of −102.11, outperforming the previous best result of −103.38 from a more complex model using 50 importance samples.
• The LVAE learns a deeper and more distributed latent representation: all five layers are active, with KL divergence more evenly distributed across layers compared to VAEs, which collapse higher layers.
• Batch normalization and deterministic warm-up are crucial for training deep stochastic models, with warm-up enabling more active units and preventing premature collapse.
• PCA visualizations show that LVAE maintains structured, class-discriminative representations even in the topmost latent layer, while VAEs collapse higher layers to the prior.
• The LVAE provides a tighter bound on the true log-likelihood than standard VAEs, indicating better approximation of the true posterior.