[论文解读] Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks

Lars Mescheder, Sebastian Nowozin+1 more|arXiv (Cornell University)|Jan 17, 2017

Generative Adversarial Networks and Image Synthesis参考文献 29被引用 264

一句话总结

AVB 通过引入一个辅助判别器，使对变分自编码器的推断模型具备任意表达能力，在非参数极限下统一了 VAEs 与 GANs，并具备理论保证。

ABSTRACT

Variational Autoencoders (VAEs) are expressive latent variable models that can be used to learn complex probability distributions from training data. However, the quality of the resulting model crucially relies on the expressiveness of the inference model. We introduce Adversarial Variational Bayes (AVB), a technique for training Variational Autoencoders with arbitrarily expressive inference models. We achieve this by introducing an auxiliary discriminative network that allows to rephrase the maximum-likelihood-problem as a two-player game, hence establishing a principled connection between VAEs and Generative Adversarial Networks (GANs). We show that in the nonparametric limit our method yields an exact maximum-likelihood assignment for the parameters of the generative model, as well as the exact posterior distribution over the latent variables given an observation. Contrary to competing approaches which combine VAEs with GANs, our approach has a clear theoretical justification, retains most advantages of standard Variational Autoencoders and is easy to implement.

研究动机与目标

• Enable arbitrarily expressive inference models in Variational Autoencoders using adversarial training.
• Provide theoretical guarantees that AVB recovers true posterior and maximum-likelihood parameters in the nonparametric limit.
• Demonstrate that AVB yields richer posterior approximations and competitive or state-of-the-art generative modeling results.

提出的方法

• Introduce an auxiliary discriminative network T(x, z) to approximate log qφ(z|x) − log p(z).
• Formulate a two-player game between the encoder/decoder (θ, φ) and the discriminator T to maximize the variational bound.
• Derive gradients via reparameterization when possible and show that ∇φ Eqφ(z|x)[∇φ T*(x, z)] term vanishes (Proposition 2).
• Propose Algorithm 1 (AVB) to perform stochastic gradient updates for θ, φ, and ψ (discriminator).
• Introduce Adaptive Contrast (AC) to stabilize training by contrasting qφ(z|x) with a known density rα(z|x) instead of the prior p(z).
• Provide theoretical results showing that at Nash equilibria, T*(x,z) = log qφ*(z|x) − log p(z) and (θ*, φ*) are global optima of the ELBO.

实验结果

研究问题

• RQ1 Can VAEs leverage arbitrarily expressive inference models trained with adversarial objectives while preserving max-likelihood objectives?
• RQ2 Under what conditions do AVB and its variants recover the true posterior and maximum-likelihood parameters?
• RQ3 How does AVB perform in variational inference and as a generative model compared to Gaussian-inference VAEs and GAN-based approaches?
• RQ4 Does Adaptive Contrast improve training stability and posterior quality in AVB?
• RQ5 Can AVB achieve state-of-the-art log-likelihoods on standard benchmarks like MNIST?

主要发现

• AVB enables black-box, highly expressive inference models for VAEs and improves posterior expressiveness.
• In the nonparametric limit, AVB recovers the true posterior and the true maximum-likelihood parameters for the generative model (θ*, φ*).
• AVB with Adaptive Contrast yields richer posterior distributions and often closer-to-ground-truth posteriors than Gaussian-inference VAEs, e.g., on synthetic and Eight Schools examples.
• On MNIST, AVB variants achieve competitive or state-of-the-art log-likelihood estimates compared to prior VAE approaches and other baselines when matched with appropriate decoder architectures.
• Experiment results show AVB produces sharper reconstructions and more accurate latent representations than standard VAEs with diagonal Gaussian posteriors.