[Paper Review] Towards Deeper Understanding of Variational Autoencoding Models

Shengjia Zhao, Jiaming Song+1 more|arXiv (Cornell University)|Feb 28, 2017

Generative Adversarial Networks and Image SynthesisComputer Science15 references129 citations

TL;DR

The paper introduces a general optimization framework for VAEs beyond ELBO, explains common issues like blurry samples and uninformative latent codes, and proposes sequential VAE and information-aware training to improve sample sharpness and latent utilization.

ABSTRACT

We propose a new family of optimization criteria for variational auto-encoding models, generalizing the standard evidence lower bound. We provide conditions under which they recover the data distribution and learn latent features, and formally show that common issues such as blurry samples and uninformative latent features arise when these conditions are not met. Based on these new insights, we propose a new sequential VAE model that can generate sharp samples on the LSUN image dataset based on pixel-wise reconstruction loss, and propose an optimization criterion that encourages unsupervised learning of informative latent features.

Motivation & Objective

• Provide a new general optimization criterion for variational autoencoding models beyond the standard ELBO.
• Explain why VAEs struggle with blurry samples and uninformative latent features on complex data.
• Introduce methods to alleviate these issues, including sequential VAE and informed regularization.
• Show that a more expressive inference/posterior and suitable regularization can yield sharp samples and meaningful latent features.

Proposed method

• Derive a general p_theta(x|z) with f_theta mapping latent codes to distribution parameters, and optimize a loss L not necessarily a strict ELBO.
• Propose a discriminative objective L = E_q(z)[E_q(x|z)[log p_theta(x|z)]] to avoid the trivial solution where a single p in P fits all data.
• Show that with sufficient capacity, the model can learn q_phi(z|x) such that P_{f_theta(z)} approximates q(x|z) for each z.
• Introduce a regularized VAE L_VAE = E_{q_phi(x,z)}[log p_theta(x|z)] - R(q_phi) and relate R to the ELBO when R corresponds to KL divergence terms.
• Present sequential VAE and infusion-training-inspired augmentation to progressively simplify the posterior q(x|z) and generate sharper samples.
• Demonstrate that unregularized VAE can retain informative latent features, whereas ELBO tends to suppress them, and that complex posteriors can be better handled by sequential augmentation.

Experimental results

Research questions

• RQ1 Can a general optimization framework for VAEs recover the data distribution under suitable conditions without relying on the traditional ELBO?
• RQ2 Why do VAEs produce blurry samples and why is the latent code often ignored, and how can we design training to mitigate these issues?
• RQ3 How can we leverage more expressive inference/posterior families and sequential augmentation to achieve sharp samples on complex datasets like LSUN?
• RQ4 Under what conditions does regularized versus unregularized training affect latent code utilization and sampling quality?
• RQ5 What role does the choice of P (the family of p(x|z)) play in the posterior q(x|z) and the resulting sample fidelity?

Key findings

• A generalized VAEs framework can learn data distributions using a discriminative objective that varies q(x|z) across z, enabling use of richer posteriors.
• Blurry samples arise when the posterior q(x|z) is too complex for a simple P; using sequential VAE or infusion-inspired augmentation reduces posterior variance and yields sharper samples.
• ELBO-based training can drive latent information to zero, whereas unregularized training preserves mutual information between x and z, preserving informative latent codes.
• Sequential VAE with progressively richer inference and reconstruction steps improves sample sharpness on LSUN using 2-norm pixel loss.
• When P is complex (e.g., PixelCNN), ELBO may still lead to latent-code neglect; removing strict regularization can maintain latent information while achieving competitive sample quality.
• The framework provides a principled way to diagnose and mitigate common VAE pathologies and to design training schemes that maintain informative latent features.