[Paper Review] Variational Approaches for Auto-Encoding Generative Adversarial Networks

Mihaela Rosca, Balaji Lakshminarayanan+2 more|arXiv (Cornell University)|Jun 15, 2017

Generative Adversarial Networks and Image Synthesis40 references247 citations

TL;DR

The paper proposes a principled hybrid of variational auto-encoders and GANs (alpha-GAN) by using synthetic likelihoods and implicit posterior distributions learned via discriminators, aiming to combine VAE stability with GANs’ sharpness, and evaluates against DC-GAN, WGAN-GP, and AGE.

ABSTRACT

Auto-encoding generative adversarial networks (GANs) combine the standard GAN algorithm, which discriminates between real and model-generated data, with a reconstruction loss given by an auto-encoder. Such models aim to prevent mode collapse in the learned generative model by ensuring that it is grounded in all the available training data. In this paper, we develop a principle upon which auto-encoders can be combined with generative adversarial networks by exploiting the hierarchical structure of the generative model. The underlying principle shows that variational inference can be used a basic tool for learning, but with the in- tractable likelihood replaced by a synthetic likelihood, and the unknown posterior distribution replaced by an implicit distribution; both synthetic likelihoods and implicit posterior distributions can be learned using discriminators. This allows us to develop a natural fusion of variational auto-encoders and generative adversarial networks, combining the best of both these methods. We describe a unified objective for optimization, discuss the constraints needed to guide learning, connect to the wide range of existing work, and use a battery of tests to systematically and quantitatively assess the performance of our method.

Motivation & Objective

• Motivate and formalize a principled fusion of auto-encoders with GANs using the hierarchical latent structure of GANs.
• Develop a variational framework where intractable likelihoods are replaced by synthetic likelihoods and posterior distributions by implicit discriminators.
• Propose a unified alpha-GAN objective that combines reconstruction, synthetic likelihood, and latent-code discrimination.
• Provide practical training strategies to stabilize learning and prevent mode collapse while enabling reconstructions.

Proposed method

• Use density-ratio tricks and discriminators to perform variational inference with implicit posteriors.
• Replace intractable likelihoods with synthetic likelihoods learned via a discriminator-based density ratio.
• Introduce an explicit reconstruction term with an L1 loss and an adversarial reconstruction term via a synthetic likelihood discriminator.
• Incorporate a latent-space discriminator to align encoder outputs with the prior distribution.
• Formulate a hybrid objective that combines reconstruction, synthetic likelihood, and latent-code matching (alpha-GAN).
• Provide training enhancements: reverse KL updates for non-saturating gradients and sharing a single discriminator for fake data and reconstructions.

Experimental results

Research questions

• RQ1 Can variational inference be effectively integrated with GANs by using implicit posteriors and synthetic likelihoods?
• RQ2 Does combining an auto-encoder reconstruction loss with adversarial training improve mode coverage and sample quality without pre-training?
• RQ3 How does alpha-GAN compare to DC-GAN, WGAN-GP, and AGE across standard datasets in reconstruction and generation tasks?
• RQ4 What practical training strategies stabilize alpha-GAN optimization and support faithful reconstructions?

Key findings

• Alpha-GAN achieves competitive results with state-of-the-art GAN variants and AGE across several datasets.
• The model provides reconstructions of inputs, addressing a key limitation of vanilla GANs.
• Alpha-GAN’s evaluation uses a battery of metrics including independent Wasserstein critic and inception scores to assess sample quality and mode coverage.
• Reconstruction terms help mitigate mode collapse while adversarial components improve sample realism.
• Architectural choices and update ratios between generator, encoder, and discriminators significantly influence performance and stability.
• Comparisons highlight nontrivial differences between image quality metrics when different evaluation networks are used (ImageNet vs CIFAR-10 nets).