[Paper Review] Variational Lossy Autoencoder

Xi Chen, Diederik P. Kingma+6 more|arXiv (Cornell University)|Nov 8, 2016

Generative Adversarial Networks and Image Synthesis257 citations

TL;DR

This paper introduces Variational Lossy Autoencoder (VLAE), a VAE model augmented with autoregressive priors and decoders to control information placement, enabling lossy, global representations and state-of-the-art density estimates on several image datasets.

ABSTRACT

Representation learning seeks to expose certain aspects of observed data in a learned representation that's amenable to downstream tasks like classification. For instance, a good representation for 2D images might be one that describes only global structure and discards information about detailed texture. In this paper, we present a simple but principled method to learn such global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN. Our proposed VAE model allows us to have control over what the global latent code can learn and , by designing the architecture accordingly, we can force the global latent code to discard irrelevant information such as texture in 2D images, and hence the VAE only "autoencodes" data in a lossy fashion. In addition, by leveraging autoregressive models as both prior distribution $p(z)$ and decoding distribution $p(x|z)$, we can greatly improve generative modeling performance of VAEs, achieving new state-of-the-art results on MNIST, OMNIGLOT and Caltech-101 Silhouettes density estimation tasks.

Motivation & Objective

• Aim to learn global, disentangled representations by forcing the latent code to capture long-range structure while local details are modeled by an autoregressive decoder.
• Investigate why VAEs with powerful decoders often ignore latent variables and how to mitigate this by information placement strategies.
• Propose two complementary enhancements—explicit information placement via constrained autoregressive decoding and a learned autoregressive prior—to improve both representations and density estimation.
• Evaluate VLAE on binary image datasets to assess lossy coding of global structure and assess density estimation performance.
• Demonstrate improvements over prior VAEs and competitive results on CIFAR-10 with autoregressive components.

Proposed method

• Combine Variational Autoencoders with neural autoregressive models to form VLAE.
• Use a decoding distribution p(x|z) with a small receptive field to force global information into z, enabling lossy representations of data.
• Constrain the autoregressive decoder so it can model only local statistics, pushing long-range structure into the latent code.
• Parametrize the prior p(z) with an autoregressive flow (AF), which is equivalent to an inverse autoregressive flow (IAF) posterior in the encoder path, yielding a more expressive generative model with no extra training cost.
• Optionally employ an autoregressive decoder (PixelCNN) to improve density estimation.
• Leverage Bits-Back Coding interpretation to analyze when latent code z is utilized and to motivate information-placement strategies.

Experimental results

Research questions

• RQ1 Can VLAE learn lossy codes that encode global statistics in images?
• RQ2 Does replacing an IAF posterior with an AF prior improve density estimation and latent utilization?
• RQ3 Does using an autoregressive decoding distribution improve density estimation performance compared to non-autoregressive decoders?
• RQ4 How does the receptive-field size of the autoregressive decoder affect what information is stored in z?
• RQ5 Is VLAE competitive with or superior to prior state-of-the-art density models on standard image benchmarks (MNIST, OMNIGLOT, Caltech silhouettes, CIFAR-10)?

Key findings

• VLAE learns lossy representations where global structure is captured in the latent code, while local textures are modeled by the autoregressive decoder.
• AF prior outperforms an equivalent IAF posterior in terms of negative log-likelihood on statically binarized MNIST, indicating the deeper generative model from AF prior is beneficial.
• Using an autoregressive decoder (PixelCNN) with an AF prior yields state-of-the-art or competitive density estimates on MNIST, OMNIGLOT, and Caltech-101 Silhouettes datasets.
• On CIFAR-10, VLAE with DenseNet backbones achieves competitive likelihoods, outperforming several prior variational models and approaching PixelCNN++-level performance.
• The receptive-field size of the PixelCNN decoder influences the extent to which the latent code encodes global versus local information; smaller fields preserve more local detail in z, larger fields transfer more structure to z.