[Paper Review] Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse

James Lucas, George Tucker+2 more|arXiv (Cornell University)|Nov 6, 2019

Generative Adversarial Networks and Image SynthesisComputer Science32 references18 citations

TL;DR

This paper challenges the common belief that the ELBO objective causes posterior collapse in VAEs by analyzing linear VAEs and showing they exactly recover pPCA solutions. It proves the ELBO introduces no spurious local maxima beyond those in log marginal likelihood, and demonstrates that observation noise is critical for escaping collapsed solutions—insights that extend to non-linear VAEs.

ABSTRACT

Posterior collapse in Variational Autoencoders (VAEs) arises when the variational posterior distribution closely matches the prior for a subset of latent variables. This paper presents a simple and intuitive explanation for posterior collapse through the analysis of linear VAEs and their direct correspondence with Probabilistic PCA (pPCA). We explain how posterior collapse may occur in pPCA due to local maxima in the log marginal likelihood. Unexpectedly, we prove that the ELBO objective for the linear VAE does not introduce additional spurious local maxima relative to log marginal likelihood. We show further that training a linear VAE with exact variational inference recovers an identifiable global maximum corresponding to the principal component directions. Empirically, we find that our linear analysis is predictive even for high-capacity, non-linear VAEs and helps explain the relationship between the observation noise, local maxima, and posterior collapse in deep Gaussian VAEs.

Motivation & Objective

• To challenge the widespread assumption that the ELBO objective causes posterior collapse in VAEs.
• To analyze linear VAEs as a tractable model to understand the root causes of posterior collapse.
• To investigate whether the ELBO introduces spurious local maxima that lead to collapsed posteriors.
• To determine the role of observation noise in escaping suboptimal solutions during VAE training.
• To extend insights from linear models to high-capacity non-linear VAEs.

Proposed method

• Analyzes linear VAEs with diagonal variational posterior to establish a direct correspondence with probabilistic PCA (pPCA).
• Proves that the global optimum of the ELBO in linear VAEs recovers the true principal components, not just their subspace.
• Shows that the ELBO objective for linear VAEs has no additional spurious local maxima compared to log marginal likelihood optimization.
• Uses theoretical analysis of pPCA stationary points to link observation noise variance with stability of collapsed solutions.
• Empirically evaluates deep Gaussian VAEs on MNIST and CelebA, varying observation noise and measuring posterior collapse via KL divergence.
• Applies KL-annealing and tracks latent dimension collapse across training to validate theoretical predictions.

Experimental results

Research questions

• RQ1 Does the ELBO objective introduce spurious local maxima that cause posterior collapse in linear VAEs?
• RQ2 Can linear VAEs recover the exact principal components, and is this solution globally optimal under the ELBO?
• RQ3 How does observation noise influence the stability of local maxima corresponding to posterior collapse?
• RQ4 Do insights from linear VAEs generalize to high-capacity non-linear VAEs?
• RQ5 Can controlling observation noise help escape collapsed solutions in deep VAEs?

Key findings

• The ELBO for linear VAEs does not introduce any spurious local maxima beyond those in log marginal likelihood optimization.
• The global optimum of the linear VAE ELBO recovers the exact principal component directions, not just the subspace.
• Posterior collapse in pPCA arises from local maxima in log marginal likelihood, not from the ELBO objective.
• Observation noise plays a crucial role in escaping collapsed solutions—reducing noise helps escape suboptimal local maxima.
• Empirical results on deep Gaussian VAEs show that lower observation noise correlates with reduced posterior collapse.
• On CelebA, when observation noise is fixed at 0.001, posterior collapse reaches 2680.2% (mean over 5 runs), but tuning noise reduces collapse to 483.8%.