[Paper Review] BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling

Lars Maaløe, M. Fraccaro+2 more|arXiv (Cornell University)|Feb 6, 2019

Generative Adversarial Networks and Image SynthesisComputer Science60 references71 citations

TL;DR

BIVA introduces a very deep hierarchy of stochastic latent variables with a bidirectional inference network and a deterministic top-down path, achieving strong likelihoods, sharp image generation, anomaly detection, and semi-supervised classification.

ABSTRACT

With the introduction of the variational autoencoder (VAE), probabilistic latent variable models have received renewed attention as powerful generative models. However, their performance in terms of test likelihood and quality of generated samples has been surpassed by autoregressive models without stochastic units. Furthermore, flow-based models have recently been shown to be an attractive alternative that scales well to high-dimensional data. In this paper we close the performance gap by constructing VAE models that can effectively utilize a deep hierarchy of stochastic variables and model complex covariance structures. We introduce the Bidirectional-Inference Variational Autoencoder (BIVA), characterized by a skip-connected generative model and an inference network formed by a bidirectional stochastic inference path. We show that BIVA reaches state-of-the-art test likelihoods, generates sharp and coherent natural images, and uses the hierarchy of latent variables to capture different aspects of the data distribution. We observe that BIVA, in contrast to recent results, can be used for anomaly detection. We attribute this to the hierarchy of latent variables which is able to extract high-level semantic features. Finally, we extend BIVA to semi-supervised classification tasks and show that it performs comparably to state-of-the-art results by generative adversarial networks.

Motivation & Objective

• Motivate and close the performance gap between probabilistic latent variable models and autoregressive/flow-based models in terms of test likelihood and sample quality.
• Propose a novel VAE variant—BIVA—with a deep hierarchy of stochastic latent variables, a bidirectional inference path, and a deterministic top-down pathway.
• Demonstrate the benefits of the architecture via ablation studies, and evaluate on natural and binary images, anomaly detection, and semi-supervised classification.
• Show that BIVA can perform anomaly detection using high-level semantic features learned in the latent hierarchy.

Proposed method

• Introduce a deep hierarchical VAE where each layer's latent variable is split into bottom-up and top-down components (z_i = (z_i_BU, z_i_TD)).
• Add a deterministic top-down path that provides skip connections to improve information flow and reduce latent-variable collapse.
• Use a bidirectional inference network with bottom-up stochastic and top-down stochastic paths, sharing parameters with the generative model, to form a flexible posterior q_phi(z|x).
• Factorize latent variables at each level to enable bottom-up stochastic inference and enable higher-layer covariance learning without auxiliary variables.
• Train by maximizing the ELBO with reparameterization, aided by a free-bits strategy to mitigate early latent-variable collapse; for anomaly detection, use a hierarchical lower bound L^{>k} that samples top layers from the variational posterior and lower layers from the conditional prior.
• Describe the model’s applicability to anomaly detection and a semi-supervised extension by incorporating a class variable y with a conditional x given z and y.

Experimental results

Research questions

• RQ1 Does BIVA improve test log-likelihood and sample quality beyond autoregressive and flow-based models on standard benchmarks?
• RQ2 Can the latent-variable hierarchy in BIVA capture high-level semantic features useful for anomaly detection?
• RQ3 How does the bidirectional inference path and deterministic top-down connections contribute to avoiding latent-variable collapse?
• RQ4 Can BIVA be extended to semi-supervised classification with competitive performance?
• RQ5 What is the impact of the L-level deep hierarchy and BU/TD factorization on learned representations and downstream tasks?

Key findings

• BIVA achieves state-of-the-art or competitive test likelihoods on several benchmarks, outperforming non-autoregressive latent-variable models and approaching autoregressive models in some settings.
• On binary MNIST tasks, BIVA attains strong log-likelihoods with L1 and L1e3 settings, and finetuning yields further improvements compared to prior latent-variable methods.
• For natural images, BIVA with L=15 (and variants up to L=20) delivers competitive bits-per-dimension on CIFAR-10, outperforming many flow-based and non-autoregressive models, though autoregressive models can still lead in some cases.
• A hierarchy-based anomaly detection approach using higher-level latent variables (L^{>k}) demonstrates better separation of in-distribution and out-of-distribution data than standard ELBO-based methods, indicating the model’s capacity to encode high-level semantics.
• A semi-supervised extension with a class variable y yields competitive classification performance on MNIST, comparable to contemporary GAN-based approaches.
• Qualitative samples generated from the BIVA prior (N(0, I)) on CelebA appear sharp and coherent, indicating improved generative quality over prior latent-variable models.