[Paper Review] VAE with a VampPrior

Jakub M. Tomczak, Max Welling|arXiv (Cornell University)|May 19, 2017

Generative Adversarial Networks and Image SynthesisComputer Science24 references60 citations

TL;DR

The paper introduces VampPrior, a variational posterior–driven prior (mixture of posteriors on learnable pseudo-inputs), and a two-layer hierarchical VAE that jointly learns prior and posterior to improve latent representations and avoid inactive latent units, achieving strong results on six datasets.

ABSTRACT

Many different methods to train deep generative models have been introduced in the past. In this paper, we propose to extend the variational auto-encoder (VAE) framework with a new type of prior which we call "Variational Mixture of Posteriors" prior, or VampPrior for short. The VampPrior consists of a mixture distribution (e.g., a mixture of Gaussians) with components given by variational posteriors conditioned on learnable pseudo-inputs. We further extend this prior to a two layer hierarchical model and show that this architecture with a coupled prior and posterior, learns significantly better models. The model also avoids the usual local optima issues related to useless latent dimensions that plague VAEs. We provide empirical studies on six datasets, namely, static and binary MNIST, OMNIGLOT, Caltech 101 Silhouettes, Frey Faces and Histopathology patches, and show that applying the hierarchical VampPrior delivers state-of-the-art results on all datasets in the unsupervised permutation invariant setting and the best results or comparable to SOTA methods for the approach with convolutional networks.

Motivation & Objective

• Motivate and improve VAEs by making the prior more flexible and better aligned with the variational posterior.
• Introduce VampPrior: a multimodal prior formed from posteriors conditioned on learnable pseudo-inputs.
• Propose a two-layer hierarchical VAE to mitigate inactive latent variables and enhance representation quality.
• Demonstrate empirical gains over standard normal priors across diverse datasets and architectures.

Proposed method

• Derive the ELBO for VAEs and reinterpret it as a reconstruction term plus regularization components.
• Propose VampPrior: p(z) = (1/K) sum_k q_phi(z|u_k), where u_k are learnable pseudo-inputs tied to the encoder q_phi.
• Couple the prior with the posterior to encourage joint optimization and richer latent structures.
• Extend to a two-layer VAE with z1 and z2, where p(z1|z2) and p(z2) follow VampPrior and q_phi(z1|x,z2), q_psi(z2|x) define the approximate posteriors.
• Explore alternative priors (SG, MoG, VampPrior data) for ablation studies.
• Evaluate with MLP and CNN-based encoders/decoders, including PixelCNN decoders, on six image datasets.

Experimental results

Research questions

• RQ1 Does the VampPrior improve the latent representation quality and generative performance compared to the standard normal prior?
• RQ2 Does a two-layer hierarchical VAE with VampPrior mitigate inactive stochastic units and outperform one-layer VAEs across datasets?
• RQ3 How does VampPrior compare to a mixture of Gaussians prior and to data-fixed pseudo-inputs in terms of performance and learning dynamics?
• RQ4 What is the impact of using convolutional architectures and autoregressive decoders with VampPrior on benchmark image datasets?

Key findings

• VampPrior repeatedly improves test log-likelihood over the standard normal prior across multiple architectures and datasets.
• The hierarchical VampPrior VAE achieves state-of-the-art results on static MNIST, dynamic MNIST, OMNIGLOT, Caltech 101 Silhouettes, and competitive results on others, particularly with powerful decoders.
• VampPrior increases the number of active stochastic units, especially at the second layer, addressing the inactive unit problem in deeper VAEs.
• Learned pseudo-inputs tend to collapse into prototypical objects, yet generate diverse feature variants, indicating meaningful and non-memorized representations.
• Compared to MoG and SG priors, VampPrior with coupled posterior yields superior ELBO and generation quality; random pseudo-inputs (VampPrior data) offer less flexibility.
• Using convolutional encoders/decoders and PixelCNN decoders with VampPrior yields some of the best results among non-autoregressive and autoregressive VAEs on the tested datasets.