[Paper Review] Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks

Lars Mescheder, Sebastian Nowozin+1 more|arXiv (Cornell University)|Jan 17, 2017

Generative Adversarial Networks and Image SynthesisComputer Science29 references264 citations

TL;DR

AVB enables training VAEs with arbitrarily expressive inference models by introducing an auxiliary discriminator, unifying VAEs and GANs with theoretical guarantees in the nonparametric limit.

ABSTRACT

Variational Autoencoders (VAEs) are expressive latent variable models that can be used to learn complex probability distributions from training data. However, the quality of the resulting model crucially relies on the expressiveness of the inference model. We introduce Adversarial Variational Bayes (AVB), a technique for training Variational Autoencoders with arbitrarily expressive inference models. We achieve this by introducing an auxiliary discriminative network that allows to rephrase the maximum-likelihood-problem as a two-player game, hence establishing a principled connection between VAEs and Generative Adversarial Networks (GANs). We show that in the nonparametric limit our method yields an exact maximum-likelihood assignment for the parameters of the generative model, as well as the exact posterior distribution over the latent variables given an observation. Contrary to competing approaches which combine VAEs with GANs, our approach has a clear theoretical justification, retains most advantages of standard Variational Autoencoders and is easy to implement.

Motivation & Objective

• Enable arbitrarily expressive inference models in Variational Autoencoders using adversarial training.
• Provide theoretical guarantees that AVB recovers true posterior and maximum-likelihood parameters in the nonparametric limit.
• Demonstrate that AVB yields richer posterior approximations and competitive or state-of-the-art generative modeling results.

Proposed method

• Introduce an auxiliary discriminative network T(x, z) to approximate log qφ(z|x) − log p(z).
• Formulate a two-player game between the encoder/decoder (θ, φ) and the discriminator T to maximize the variational bound.
• Derive gradients via reparameterization when possible and show that ∇φ Eqφ(z|x)[∇φ T*(x, z)] term vanishes (Proposition 2).
• Propose Algorithm 1 (AVB) to perform stochastic gradient updates for θ, φ, and ψ (discriminator).
• Introduce Adaptive Contrast (AC) to stabilize training by contrasting qφ(z|x) with a known density rα(z|x) instead of the prior p(z).
• Provide theoretical results showing that at Nash equilibria, T*(x,z) = log qφ*(z|x) − log p(z) and (θ*, φ*) are global optima of the ELBO.

Experimental results

Research questions

• RQ1 Can VAEs leverage arbitrarily expressive inference models trained with adversarial objectives while preserving max-likelihood objectives?
• RQ2 Under what conditions do AVB and its variants recover the true posterior and maximum-likelihood parameters?
• RQ3 How does AVB perform in variational inference and as a generative model compared to Gaussian-inference VAEs and GAN-based approaches?
• RQ4 Does Adaptive Contrast improve training stability and posterior quality in AVB?
• RQ5 Can AVB achieve state-of-the-art log-likelihoods on standard benchmarks like MNIST?

Key findings

• AVB enables black-box, highly expressive inference models for VAEs and improves posterior expressiveness.
• In the nonparametric limit, AVB recovers the true posterior and the true maximum-likelihood parameters for the generative model (θ*, φ*).
• AVB with Adaptive Contrast yields richer posterior distributions and often closer-to-ground-truth posteriors than Gaussian-inference VAEs, e.g., on synthetic and Eight Schools examples.
• On MNIST, AVB variants achieve competitive or state-of-the-art log-likelihood estimates compared to prior VAE approaches and other baselines when matched with appropriate decoder architectures.
• Experiment results show AVB produces sharper reconstructions and more accurate latent representations than standard VAEs with diagonal Gaussian posteriors.