[Paper Review] Lossless compression with state space models using bits back coding

James T. Townsend, Iain Murray|arXiv (Cornell University)|Mar 18, 2021

Gaussian Processes and Bayesian Inference16 references4 citations

TL;DR

This paper introduces IconoCLaSM, a novel method for lossless compression of sequential data using state space models (SSMs) by interleaving bits-back coding with time steps, achieving compression rates near the model's ELBO with only O(1) initial overhead. The approach enables efficient, scalable compression for time-series data like video when combined with deep latent variable models.

ABSTRACT

We generalize the 'bits back with ANS' method to time-series models with a latent Markov structure. This family of models includes hidden Markov models (HMMs), linear Gaussian state space models (LGSSMs) and many more. We provide experimental evidence that our method is effective for small scale models, and discuss its applicability to larger scale settings such as video compression.

Motivation & Objective

• To address the high initial bits overhead in standard bits-back coding when compressing sequential data with latent Markov structures.
• To extend the bits-back with ANS (BB-ANS) framework to state space models (SSMs), which model sequences through latent Markov chains.
• To enable practical, scalable lossless compression of long sequences by reducing the initial overhead to O(1) through interleaved encoding and decoding steps.
• To demonstrate that compression rates converge to the model's ELBO for long sequences, making the method suitable for large-scale applications like video compression.
• To provide a general framework applicable to HMMs, LGSSMs, and deep latent variable models with tractable posterior conditionals.

Proposed method

• Proposes IconoCLaSM, a method that interleaves bits-back coding steps with time steps in a state space model to reduce initial overhead.
• Requires a factorized approximate posterior: $ Q(z_{1}, ext{...},z_{T}|x_{1}, ext{...},x_{T}) = Q(z_{T}|x_{1}, ext{...},x_{T}) \prod_{t=1}^{T-1} Q(z_{t}|x_{1}, ext{...},x_{t},z_{t+1}) $, enabling backward sampling.
• Uses ANS encoding and decoding with push/pop operations on conditional distributions: $ x_t \rightarrow P(x_t|z_t) $, $ z_t \leftarrow Q(z_t|x_{1..t}, z_{t+1}) $, $ z_t \rightarrow P(z_t|z_{t-1}) $.
• Employs a reverse encoding order: starting from $ z_T $, then $ x_T $, $ z_{T-1} $, ..., $ x_1 $, $ z_1 $, to enable bits-back during encoding.
• Decoding proceeds in forward order: $ z_1 \leftarrow P(z_1) $, $ x_1 \leftarrow P(x_1|z_1) $, ..., $ z_T \rightarrow Q(z_T|x_{1..T}) $, reconstructing the sequence.
• Leverages message-passing or RNN-based inference to compute required posterior conditionals, enabling application to HMMs and deep SSMs.

Experimental results

Research questions

• RQ1 Can bits-back coding be generalized to sequential models with latent Markov structure to achieve near-optimal compression rates with minimal initial overhead?
• RQ2 What factorization of the approximate posterior is required to enable interleaved bits-back coding in state space models?
• RQ3 Does the compression rate of IconoCLaSM converge to the ELBO for long sequences, as in vanilla BB-ANS?
• RQ4 Can IconoCLaSM be applied to real-world sequential data such as text or video when combined with deep latent variable models?
• RQ5 Is it feasible to extend IconoCLaSM to hierarchical models or combine it with Monte Carlo methods to improve the ELBO?

Key findings

• For long sequences, the compression rate of IconoCLaSM converges to the information content $ h(x) $, achieving near-optimal compression.
• In experiments with a small HMM, the ratio of compressed message length to information content $ l(m)/h(x) $ tends to 1 as sequence length $ T $ increases.
• The method achieves compression rates close to the ELBO for both synthetic data (exact samples from an HMM) and real data (War and Peace text).
• The initial bits overhead diminishes with increasing sequence length, demonstrating O(1) scaling of overhead, which is critical for scalability.
• The method is effective even when the model is not perfectly fitted to the data, as shown in compression of English text using an HMM trained on a different text segment.
• The framework is general and applicable to HMMs, linear Gaussian SSMs, and deep latent variable models, provided the required posterior conditionals are available.