[Paper Review] The continuous Bernoulli: fixing a pervasive error in variational autoencoders

Gabriel Loaiza-Ganem, John P. Cunningham|arXiv (Cornell University)|Jul 16, 2019

Generative Adversarial Networks and Image SynthesisComputer Science22 citations

TL;DR

This paper introduces the continuous Bernoulli distribution, a new [0,1]-supported likelihood for variational autoencoders (VAEs), to correct the pervasive error of modeling [0,1]-valued pixel data with a discrete Bernoulli likelihood. By replacing the standard Bernoulli with the continuous Bernoulli, the method improves sample quality and reconstruction performance across multiple datasets, demonstrating that this seemingly minor correction yields substantial quantitative and qualitative gains in VAE training.

ABSTRACT

Variational autoencoders (VAE) have quickly become a central tool in machine learning, applicable to a broad range of data types and latent variable models. By far the most common first step, taken by seminal papers and by core software libraries alike, is to model MNIST data using a deep network parameterizing a Bernoulli likelihood. This practice contains what appears to be and what is often set aside as a minor inconvenience: the pixel data is [0,1] valued, not {0,1} as supported by the Bernoulli likelihood. Here we show that, far from being a triviality or nuisance that is convenient to ignore, this error has profound importance to VAE, both qualitative and quantitative. We introduce and fully characterize a new [0,1]-supported, single parameter distribution: the continuous Bernoulli, which patches this pervasive bug in VAE. This distribution is not nitpicking; it produces meaningful performance improvements across a range of metrics and datasets, including sharper image samples, and suggests a broader class of performant VAE.

Motivation & Objective

• To address the widespread but problematic practice of modeling continuous [0,1]-valued pixel data in VAEs using the discrete Bernoulli likelihood.
• To identify that this mismatch between data support and likelihood support leads to significant performance degradation in VAEs.
• To propose a new, continuous, single-parameter distribution—called the continuous Bernoulli—that is properly defined on the interval [0,1].
• To demonstrate that replacing the standard Bernoulli likelihood with the continuous Bernoulli leads to measurable improvements in VAE performance.
• To establish the continuous Bernoulli as a foundational component for more accurate and performant VAE architectures.

Proposed method

• Proposes the continuous Bernoulli distribution as a proper likelihood for [0,1]-valued data, defined by a single parameter ξ ∈ (0,1).
• Derives the probability density function (PDF) of the continuous Bernoulli, which is proportional to p^ξ(1−p)^(1−ξ) for p ∈ [0,1], with a normalization constant that depends on ξ.
• Introduces a reparameterization trick for the continuous Bernoulli, enabling differentiable sampling and gradient estimation in VAEs.
• Replaces the standard Bernoulli likelihood in VAEs with the continuous Bernoulli likelihood in both the recognition and generation models.
• Employs the continuous Bernoulli in standard VAE architectures on benchmark datasets such as MNIST, CIFAR-10, and CelebA, using standard training protocols.
• Validates the method through ablation studies and comparisons against standard VAEs using discrete Bernoulli likelihoods.

Experimental results

Research questions

• RQ1 How does modeling [0,1]-valued pixel data with a discrete Bernoulli likelihood affect VAE performance in practice?
• RQ2 Can a continuous likelihood defined on [0,1] improve VAE sample quality and reconstruction accuracy compared to the discrete Bernoulli?
• RQ3 What is the mathematical form and normalization of a proper [0,1]-supported distribution that generalizes the Bernoulli?
• RQ4 Is there a differentiable reparameterization scheme for the proposed continuous Bernoulli distribution to enable end-to-end training in VAEs?
• RQ5 Does the continuous Bernoulli lead to consistent performance gains across diverse datasets and architectures?

Key findings

• The continuous Bernoulli distribution is a proper probability distribution on [0,1] with a single parameter, providing a natural likelihood for continuous pixel intensities.
• Replacing the discrete Bernoulli likelihood with the continuous Bernoulli in VAEs leads to sharper image samples, especially noticeable in high-resolution generation.
• The method achieves consistently better log-likelihood scores on test sets across MNIST, CIFAR-10, and CelebA compared to standard VAEs.
• The continuous Bernoulli enables more accurate posterior approximation, reducing the distributional mismatch between data and likelihood.
• The reparameterization trick for the continuous Bernoulli allows stable and efficient training of VAEs with no additional computational overhead.
• The performance gains are not marginal; they represent a fundamental correction to a long-ignored error in VAE design, with measurable improvements in both qualitative and quantitative metrics.