[Paper Review] Diagnosing and Enhancing VAE Models

Bin Dai, David Wipf|arXiv (Cornell University)|Mar 14, 2019

Generative Adversarial Networks and Image SynthesisComputer Science42 references95 citations

TL;DR

The paper analyzes Gaussian VAE assumptions, shows conditions under which ground-truth distributions can be recovered, and proposes a simple two-stage VAE enhancement that yields crisp samples and competitive FID without extra tuning.

ABSTRACT

Although variational autoencoders (VAEs) represent a widely influential deep generative model, many aspects of the underlying energy function remain poorly understood. In particular, it is commonly believed that Gaussian encoder/decoder assumptions reduce the effectiveness of VAEs in generating realistic samples. In this regard, we rigorously analyze the VAE objective, differentiating situations where this belief is and is not actually true. We then leverage the corresponding insights to develop a simple VAE enhancement that requires no additional hyperparameters or sensitive tuning. Quantitatively, this proposal produces crisp samples and stable FID scores that are actually competitive with a variety of GAN models, all while retaining desirable attributes of the original VAE architecture. A shorter version of this work will appear in the ICLR 2019 conference proceedings (Dai and Wipf, 2019). The code for our model is available at https://github.com/daib13/ TwoStageVAE.

Motivation & Objective

• Investigate how Gaussian encoder/decoder assumptions impact the ability of VAEs to recover ground-truth distributions under different manifold dimensions.
• Characterize optimal VAE solutions and the role of the latent dimensionality in learning the data manifold versus the distribution within the manifold.
• Propose a practical two-stage VAE enhancement to address non-uniqueness and improve sample quality without additional tuning.
• Demonstrate that the two-stage approach can produce high-quality samples and stable FID scores under neutral testing conditions.

Proposed method

• Define a kappa-simple VAE with Gaussian encoder q_phi(z|x) and Gaussian decoder p_theta(x|z).
• Prove that when the manifold dimension equals ambient space (r=d), there exist parameter sequences that jointly optimize the VAE objective and recover the ground-truth distribution under certain conditions.
• Show that when r<d, optimal solutions can achieve minimal objective without uniquely recovering the ground-truth distribution within the manifold.
• Reveal that the VAE tends to prefer degenerate latent activity (few active dimensions) and that the reconstruction term dominates as decoder variance gamma tends to zero.
• Propose a two-stage VAE: first stage learns the low-dimensional manifold with minimal active latents, second stage learns the distribution over the latent codes to align with a standard Gaussian prior.

Experimental results

Research questions

• RQ1 Can Gaussian VAE assumptions allow recovery of the ground-truth distribution under r=d and what conditions are required?
• RQ2 What are the implications of r<d for the uniqueness of the global optimum and the learned ground-truth manifold?
• RQ3 How can we address non-uniqueness and poor latent matching to improve sample quality without extra tuning?
• RQ4 Does a simple two-stage VAE can achieve crisp samples and stable FID comparable to GANs under neutral conditions?

Key findings

• A kappa-simple VAE with kappa >= r can globally optimize the objective and recover the ground-truth distribution when r=d, given suitable conditions.
• For r<d, optimal solutions can push the objective to -infinity while matching the manifold mass but not necessarily reproduce mu_gt within the manifold.
• Optimal solutions exhibit degenerate latent activity with only r active dimensions, and the decoder variance gamma tends to zero to minimize the objective, impacting reconstructive behavior.
• Introducing a two-stage VAE can recover a good latent representation of the manifold in stage one and then learn the correct distribution on this intermediate code in stage two.
• Experiments show the two-stage approach yields crisp samples and stable FID scores that rival GANs under neutral testing without extra penalties or tuning.