[Paper Review] Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables

Friso H. Kingma, Pieter Abbeel+1 more|arXiv (Cornell University)|May 16, 2019

Advanced Data Compression Techniques35 references43 citations

TL;DR

Bit-Swap generalizes BB-ANS to hierarchical latent variable models and achieves better lossless compression rates by reducing initial bits overhead, with empirical gains over existing methods.

ABSTRACT

The bits-back argument suggests that latent variable models can be turned into lossless compression schemes. Translating the bits-back argument into efficient and practical lossless compression schemes for general latent variable models, however, is still an open problem. Bits-Back with Asymmetric Numeral Systems (BB-ANS), recently proposed by Townsend et al. (2019), makes bits-back coding practically feasible for latent variable models with one latent layer, but it is inefficient for hierarchical latent variable models. In this paper we propose Bit-Swap, a new compression scheme that generalizes BB-ANS and achieves strictly better compression rates for hierarchical latent variable models with Markov chain structure. Through experiments we verify that Bit-Swap results in lossless compression rates that are empirically superior to existing techniques. Our implementation is available at https://github.com/fhkingma/bitswap.

Motivation & Objective

• Motivate lossless compression with likelihood-based models and identify limits of BB-ANS for hierarchical latent variables.
• Introduce Bit-Swap to reduce initial-bit overhead and improve practicality for deep latent structures.
• Demonstrate empirically that Bit-Swap yields lower net bitrates on standard datasets compared to BB-ANS and benchmarks.

Proposed method

• Describe Bit-Swap as an improvement over BB-ANS for hierarchical latent variable models with a Markov chain structure.
• Leverage recursive bits-back through the chain to bound and reduce the required initial bitstream.
• Provide a formal bound showing N_init^{BitSwap} ≤ N_init^{BB-ANS} and relate to ELBO optimization.
• Specialize to hierarchical VAEs with z_L → z_{L-1} → ... → z_1 → x and define p_theta and q_theta with Markov dependencies.
• Discretize latent space and evaluate Bit-Swap against BB-ANS on MNIST, CIFAR-10, and ImageNet (32×32).
• Report cumulative moving averages and net bitrates to demonstrate amortization and practical performance.

Experimental results

Research questions

• RQ1 Can Bit-Swap reduce the initial bits overhead compared to BB-ANS for hierarchical latent variable models?
• RQ2 Do hierarchical latent variable models with Markov structure yield superior compression rates when paired with Bit-Swap versus BB-ANS and baselines?
• RQ3 How do Bit-Swap and BB-ANS scale with latent depth L in terms of compression efficiency and amortization of initial bits?
• RQ4 What is the empirical compression performance of Bit-Swap on MNIST, CIFAR-10, and ImageNet (32×32 scale) compared to existing schemes?
• RQ5 How close do Bit-Swap results come to the negative ELBO in practice across datasets?

Key findings

• Bit-Swap reduces the initial bitstream requirement and outperforms BB-ANS on hierarchical latent variable models.
• The cumulative moving average of net bitrate for Bit-Swap converges toward the negative ELBO, with faster amortization than BB-ANS as depth L increases.
• Bit-Swap achieves lower average net bitrate than BB-ANS across MNIST, CIFAR-10, and ImageNet (32×32) for multiple depths.
• Bit-Swap outperforms conventional lossless codecs (Gzip, bzip2, LZMA, PNG, WebP) and BB-ANS on the tested datasets.
• Bit-Swap reduces to BB-ANS when L=1, preserving correctness in the single-layer case.
• The model-based results show hierarchical latent variable models improve density estimation and compression efficiency when paired with Bit-Swap.