[Paper Review] Practical Lossless Compression with Latent Variables using Bits Back Coding

James T. Townsend, Thomas G. Bird+1 more|arXiv (Cornell University)|Jan 15, 2019

Generative Adversarial Networks and Image Synthesis29 references46 citations

TL;DR

BB-ANS enables lossless compression with latent variable models by using bits-back coding with ANS, demonstrated by MNIST compression with a VAE at rates close to the negative ELBO.

ABSTRACT

Deep latent variable models have seen recent success in many data domains. Lossless compression is an application of these models which, despite having the potential to be highly useful, has yet to be implemented in a practical manner. We present `Bits Back with ANS' (BB-ANS), a scheme to perform lossless compression with latent variable models at a near optimal rate. We demonstrate this scheme by using it to compress the MNIST dataset with a variational auto-encoder model (VAE), achieving compression rates superior to standard methods with only a simple VAE. Given that the scheme is highly amenable to parallelization, we conclude that with a sufficiently high quality generative model this scheme could be used to achieve substantial improvements in compression rate with acceptable running time. We make our implementation available open source at https://github.com/bits-back/bits-back .

Motivation & Objective

• Motivate lossless compression as an application of probabilistic latent-variable models.
• Develop a practical coding scheme (BB-ANS) that handles latent variables efficiently.
• Demonstrate near-ELBO compression rates on MNIST using a VAE.
• Address discretization of continuous latents and the need for pristine seed bits.
• Highlight parallelizability and potential for scaling with better generative models.

Proposed method

• Introduce bits back coding and its equivalence to negative ELBO as the optimal coding cost.
• Replace traditional AC with ANS to enable seamless chaining in BB-ANS.
• Discretize continuous latent spaces via equal-mass bucketing under the prior to enable finite-alphabet coding.
• Show how to seed the BB-ANS chain with a supply of clean bits and perform chaining across data points.
• Train a VAE with Gaussian prior and Gaussian approximate posterior, and use a beta-binomial output for MNIST pixels.
• Demonstrate that compression rate closely tracks the negative ELBO observed on test data.

Experimental results

Research questions

• RQ1 Can BB-ANS achieve compression rates near the negative ELBO for latent variable models on practical datasets?
• RQ2 How can continuous latent variables be discretized without significant loss in coding efficiency?
• RQ3 What is the impact of finite-precision, discretization, and non-clean seed bits on BB-ANS performance?
• RQ4 How does BB-ANS compare with standard generic compression algorithms on MNIST, and how might it scale with state-of-the-art models?

Key findings

• BB-ANS compresses MNIST data to rates superior to bz2, gzip, PNG, and WebP for both binarized and full MNIST using a simple VAE.
• Compression rates achieved by BB-ANS are very close to the negative ELBO of the trained VAE (small ~1% discrepancy).
• Continuous latent variables can be coded with bits back up to arbitrary precision without affecting the coding rate, enabling BB-ANS with VAEs.
• The method is compatible with parallelization and could yield substantial gains with higher-quality generative models.
• The authors provide an open-source implementation and argue BB-ANS could outperform generic compression schemes when paired with state-of-the-art latent variable models.