[Paper Review] DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors

Arash Vahdat, Evgeny Andriyash+1 more|arXiv (Cornell University)|May 1, 2018

Generative Adversarial Networks and Image Synthesis21 citations

TL;DR

This paper proposes relaxed Boltzmann priors for discrete variational autoencoders using generalized overlapping transformations and the Gaussian integral trick, enabling training with tighter importance-weighted bounds. Experiments on MNIST and OMNIGLOT show improved generation quality over prior discrete VAEs with Boltzmann priors.

ABSTRACT

Boltzmann machines are powerful distributions that have been shown to be an effective prior over binary latent variables in variational autoencoders (VAEs). However, previous methods for training discrete VAEs have used the evidence lower bound and not the tighter importance-weighted bound. We propose two approaches for relaxing Boltzmann machines to continuous distributions that permit training with importance-weighted bounds. These relaxations are based on generalized overlapping transformations and the Gaussian integral trick. Experiments on the MNIST and OMNIGLOT datasets show that these relaxations outperform previous discrete VAEs with Boltzmann priors. An implementation which reproduces these results is available.

Motivation & Objective

• To address the limitation of previous discrete VAEs using only the evidence lower bound (ELBO) instead of tighter importance-weighted bounds.
• To develop continuous relaxations of Boltzmann machines that preserve their expressive power while enabling differentiable training.
• To enable training discrete VAEs with importance-weighted bounds by transforming discrete latent distributions into continuous, differentiable approximations.
• To improve sample quality and likelihood estimates in discrete VAEs by combining expressive priors with tighter variational bounds.

Proposed method

• Proposes generalized overlapping transformations to relax discrete Boltzmann distributions into continuous, differentiable forms.
• Applies the Gaussian integral trick to derive continuous relaxations that maintain the statistical properties of the original Boltzmann distributions.
• Uses the relaxed distributions as priors in a VAE framework, enabling backpropagation through discrete latent variables.
• Employs importance-weighted autoencoders (IWAE) with the relaxed priors to tighten the variational lower bound and improve model capacity.
• Derives a differentiable objective that combines the relaxed prior with the likelihood of the data under the decoder.
• Provides an implementation that reproduces the reported results on MNIST and OMNIGLOT.

Experimental results

Research questions

• RQ1 Can continuous relaxations of Boltzmann priors enable training discrete VAEs with importance-weighted bounds?
• RQ2 How do the proposed relaxations compare to standard ELBO-based training in terms of likelihood and sample quality?
• RQ3 Do the generalized overlapping transformations and Gaussian integral trick preserve the expressiveness of Boltzmann priors while enabling differentiable optimization?
• RQ4 What is the impact of using tighter bounds on the performance of discrete VAEs with structured priors?
• RQ5 Can the proposed method achieve state-of-the-art results on standard benchmark datasets like MNIST and OMNIGLOT?

Key findings

• The proposed relaxations enable training discrete VAEs with importance-weighted bounds, leading to tighter variational objectives and improved model capacity.
• Experiments on MNIST and OMNIGLOT demonstrate that the method outperforms previous discrete VAEs using Boltzmann priors with standard ELBO training.
• The use of generalized overlapping transformations and the Gaussian integral trick successfully maintains the expressive power of Boltzmann priors in a differentiable form.
• The model achieves higher likelihoods and better sample quality compared to prior approaches, indicating effective utilization of the prior structure.
• An open-source implementation is provided, confirming reproducibility of the reported results on benchmark datasets.