[Paper Review] Improving Variational Inference with Inverse Autoregressive Flow

Diederik P. Kingma, Tim Salimans+4 more|arXiv (Cornell University)|Jun 15, 2016

Generative Adversarial Networks and Image Synthesis33 references185 citations

TL;DR

Introduces inverse autoregressive flow (IAF), a scalable normalizing flow for high-dimensional latent spaces, to significantly improve variational posteriors in VAEs and achieve competitive log-likelihood on CIFAR-10 with faster sampling.

ABSTRACT

The framework of normalizing flows provides a general strategy for flexible variational inference of posteriors over latent variables. We propose a new type of normalizing flow, inverse autoregressive flow (IAF), that, in contrast to earlier published flows, scales well to high-dimensional latent spaces. The proposed flow consists of a chain of invertible transformations, where each transformation is based on an autoregressive neural network. In experiments, we show that IAF significantly improves upon diagonal Gaussian approximate posteriors. In addition, we demonstrate that a novel type of variational autoencoder, coupled with IAF, is competitive with neural autoregressive models in terms of attained log-likelihood on natural images, while allowing significantly faster synthesis.

Motivation & Objective

• Motivate and address the limitation of simple factorized posteriors in variational inference.
• Introduce a scalable normalizing flow suitable for high-dimensional latent spaces.
• Demonstrate improved posterior flexibility and tighter variational bounds.
• Showcase performance gains in deep VAE architectures on real image datasets.

Proposed method

• Propose inverse autoregressive flow (IAF) where z0 is drawn from a simple distribution and transformed through a chain of autoregressive, invertible steps.
• Each step zt = μt + σt ⊙ zt−1 with autoregressive networks producing μt, σt, and a tractable log-determinant.
• Provide a numerically stable variant using forget-gate biases inspired by LSTM updates.
• Use pixelCNN-based autoregressive networks (MADE variants) for high-dimensional latents.
• Allow reversal of variable ordering between steps to preserve volume; derive closed-form log-determinant (sum of −log σi).
• Evaluate IAF as an expressive posterior in deep VAEs on MNIST and CIFAR-10, comparing against diagonal Gaussians and other flows.

Experimental results

Research questions

• RQ1 Does inverse autoregressive flow provide scalable, flexible posterior approximations for high-dimensional latent spaces?
• RQ2 How does IAF affect the tightness of the variational lower bound and the achieved log-likelihood on standard image datasets?
• RQ3 What is the sampling speed of VAEs with IAF compared to autoregressive generative models like PixelCNN?
• RQ4 How does stacking multiple IAF transformations and using autoregressive networks affect performance on MNIST and CIFAR-10?
• RQ5 Can IAF enable competitive log-likelihood results while maintaining efficient sampling for synthesis?

Key findings

• IAF significantly improves over diagonal Gaussian posteriors in variational autoencoders.
• Deeper and wider IAF posteriors yield tighter variational bounds and better log-likelihoods on MNIST.
• On CIFAR-10, ResNet VAE with IAF achieves 3.11 bits per dimension, competitive with state-of-the-art latent-variable models and much faster sampling than PixelCNN-based models.
• Sampling with the ResNet VAE + IAF is about 0.05 seconds per image on a Titan X, versus 52 seconds for PixelCNN-based sampling.
• Using multi-layered, autoregressive posteriors substantially tightens bounds and improves generative modeling performance compared to fixed diagonal posteriors.