[Paper Review] Distribution Matching in Variational Inference

Mihaela Rosca, Balaji Lakshminarayanan+1 more|arXiv (Cornell University)|Feb 19, 2018

Generative Adversarial Networks and Image Synthesis47 references50 citations

TL;DR

The paper shows VAEs fail to match marginal distributions in latent and visible spaces, analyzes VAE-GAN hybrids using density ratios, and concludes current hybrids have limited scalability and do not surpass GANs in sample quality.

ABSTRACT

With the increasingly widespread deployment of generative models, there is a mounting need for a deeper understanding of their behaviors and limitations. In this paper, we expose the limitations of Variational Autoencoders (VAEs), which consistently fail to learn marginal distributions in both latent and visible spaces. We show this to be a consequence of learning by matching conditional distributions, and the limitations of explicit model and posterior distributions. It is popular to consider Generative Adversarial Networks (GANs) as a means of overcoming these limitations, leading to hybrids of VAEs and GANs. We perform a large-scale evaluation of several VAE-GAN hybrids and analyze the implications of class probability estimation for learning distributions. While promising, we conclude that at present, VAE-GAN hybrids have limited applicability: they are harder to scale, evaluate, and use for inference compared to VAEs; and they do not improve over the generation quality of GANs.

Motivation & Objective

• Show that VAEs struggle to match marginal distributions in latent and visible spaces across datasets and latents.
• Analyze how explicit versus implicit distributions affect distribution matching and learning in VAEs.
• Evaluate VAE-GAN hybrids using density ratio tricks and their impact on likelihood estimates and scalability.
• Investigate the role of marginal distribution matching in latent space on learned representations.
• Assess whether VAE-GAN hybrids offer practical advantages over VAEs and GANs for generation and inference.

Proposed method

• Derive and exam the ELBO and its relation to marginal versus conditional distribution matching.
• Use experiments across ColorMNIST, CelebA, and CIFAR-10 to quantify marginal KL: q(z)||p(z) under various posteriors and visibles.
• Apply density ratio tricks to enable implicit posteriors and marginal distribution matching in latent space via Adversarial Autoencoders and AAEs.
• Introduce and compare VGH and VGH++ variants to realize marginal distribution matching in latent and visible spaces.
• Evaluate models with Inception Score, diversity, and Wasserstein critic metrics instead of ELBO.
• Analyze how density ratio estimation affects likelihood bounds and model evaluation.

Experimental results

Research questions

• RQ1 Can VAEs consistently match the marginal distributions in latent and visible spaces across datasets and latent dimensionalities?
• RQ2 Do explicit posteriors or explicit models limit marginal distribution matching more than conditional matching?
• RQ3 Do VAE-GAN hybrids improve sample quality or scalability compared to VAEs and GANs?
• RQ4 Does marginal distribution matching in latent space affect learned representations and inference capabilities?
• RQ5 Can density ratio tricks provide reliable likelihood bounds for evaluation in VAE-GAN hybrids?

Key findings

• VAEs fail to match the marginal latent posterior q(z) to the prior p(z) across datasets and latent sizes.
• Powerful explicit posteriors (e.g., RNVP) do not improve marginal distribution matching in VAEs.
• Using density ratio tricks in VAE-GAN hybrids leads to underestimation of the KL bound, harming model evaluation.
• Marginal distribution matching in latent space yields different latent representations, with AAEs learning denser representations than VAEs.
• Marginal distribution matching in visible space can improve generation quality but does not outperform pure GANs on sample quality metrics.
• VAE-GAN hybrids face scalability and hyperparameter sensitivity challenges and do not clearly outperform GANs in sample quality.