[Paper Review] InfoVAE: Information Maximizing Variational Autoencoders

Shengjia Zhao, Jiaming Song+1 more|arXiv (Cornell University)|Jun 7, 2017

Generative Adversarial Networks and Image Synthesis37 references371 citations

TL;DR

InfoVAE generalizes the VAE objective by adding a scalable KL term and a mutual information term, enabling better amortized inference and latent utilization, with MMD-based divergences yielding strong empirical performance.

ABSTRACT

A key advance in learning generative models is the use of amortized inference distributions that are jointly trained with the models. We find that existing training objectives for variational autoencoders can lead to inaccurate amortized inference distributions and, in some cases, improving the objective provably degrades the inference quality. In addition, it has been observed that variational autoencoders tend to ignore the latent variables when combined with a decoding distribution that is too flexible. We again identify the cause in existing training criteria and propose a new class of objectives (InfoVAE) that mitigate these problems. We show that our model can significantly improve the quality of the variational posterior and can make effective use of the latent features regardless of the flexibility of the decoding distribution. Through extensive qualitative and quantitative analyses, we demonstrate that our models outperform competing approaches on multiple performance metrics.

Motivation & Objective

• Motivate and diagnose failures of standard ELBO in learning and inference for VAEs.
• Propose a generalized objective that explicitly trades off data reconstruction, latent regularization, and information usage.
• Provide practical instantiations and guidance to balance X-space and Z-space losses across model families.
• Demonstrate that the proposed InfoVAE framework improves amortized inference and latent utilization across datasets and decoders.

Proposed method

• Introduce the InfoVAE objective that adds a scaling factor lambda to D_KL(q(z)||p(z)).
• Add a mutual information term I_q(x;z) to encourage informative latent representations.
• Rewrite the objective into an equivalent, optimization-friendly form (involving reconstruction term, a weighted KL(q(z|x)||p(z)), and a weighted KL(q(z)||p(z))).
• Allow replacing D_KL(q(z)||p(z)) with any strict divergence D(q(z)||p(z)) (e.g., MMD, Stein, adversarial) with preserved optimality under certain conditions.
• Show connections to and special cases of beta-VAE and Adversarial Autoencoders (AAE).
• Evaluate divergences (Adversarial, Stein, MMD) and report that MMD-regularized InfoVAE often performs best across metrics.

Experimental results

Research questions

• RQ1 Can InfoVAE alleviate the amortized inference failures observed with standard ELBO?
• RQ2 Does explicit control of information flow (I_q(x;z)) and balanced X/Z losses improve latent usage and generalization?
• RQ3 Which divergence family (MMD, Stein, adversarial) best supports the InfoVAE objective in practice?
• RQ4 How do InfoVAE variants perform on reconstruction, likelihood, and semi-supervised tasks compared to ELBO-based VAEs, beta-VAE, and AAEs?

Key findings

• ELBO optimization can lead to inaccurate amortized inference and overfitting; InfoVAE mitigates this by balancing X and Z losses and encouraging latent usage.
• InfoVAE with MMD regularization (λ large, α≈1, α=1 in some settings) achieves better or comparable log-likelihoods and sample quality across metrics.
• InfoVAE maintains meaningful latent representations even with highly flexible decoders, avoiding the information preference problem.
• Empirical results on MNIST show InfoVAE with MMD provides stable training, good posterior approximations, and strong semi-supervised performance; ELBO tends to overestimate q(z) variance.
• Table 1 shows log-likelihood estimates: ELBO 82.75, MMD-VAE 80.76, Stein-VAE 81.47, Adversarial VAE 82.21 (higher is better for this metric).
• InfoVAE variants generally outperform competing approaches across multiple metrics such as log-likelihood, sampling quality, and semi-supervised performance.