[Paper Review] IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression

Rianne van den Berg, Alexey A. Gritsenko+3 more|arXiv (Cornell University)|Jun 22, 2020

Algorithms and Data Compression52 references38 citations

TL;DR

This paper analyzes integer discrete flows (IDF) for lossless compression, refutes a claimed limitation on discrete flows, examines gradient bias, and proposes architectural improvements (IDF++) that boost compression performance with fewer flow layers.

ABSTRACT

In this paper we analyse and improve integer discrete flows for lossless compression. Integer discrete flows are a recently proposed class of models that learn invertible transformations for integer-valued random variables. Their discrete nature makes them particularly suitable for lossless compression with entropy coding schemes. We start by investigating a recent theoretical claim that states that invertible flows for discrete random variables are less flexible than their continuous counterparts. We demonstrate with a proof that this claim does not hold for integer discrete flows due to the embedding of data with finite support into the countably infinite integer lattice. Furthermore, we zoom in on the effect of gradient bias due to the straight-through estimator in integer discrete flows, and demonstrate that its influence is highly dependent on architecture choices and less prominent than previously thought. Finally, we show how different architecture modifications improve the performance of this model class for lossless compression, and that they also enable more efficient compression: a model with half the number of flow layers performs on par with or better than the original integer discrete flow model.

Motivation & Objective

• Assess the flexibility of integer discrete flows for discrete data and their capacity to model arbitrary distributions.
• Investigate the impact of straight-through gradient bias on training IDFs and compare estimators.
• Explore architectural modifications to improve IDF performance and compression efficiency.
• Quantify compression performance of IDF++ against baselines on standard datasets.

Proposed method

• Review theoretical claims about discrete flow flexibility and demonstrate embedding into Zd yields factorization capacity.
• Analyze gradient bias via straight-through estimator and compare with unbiased gradients on CIFAR-10.
• Experiment with architecture choices (coupling layers, DenseNet blocks, normalization) to improve IDF performance.
• Introduce IDF++ modifications: invert channel permutations after each layer, learnable scaling for bijectors (rezero), group normalization, Swish activations, and DenseNet++ blocks.
• Train and evaluate on CIFAR-10, ImageNet-32, ImageNet-64; compare against IDF and hand-designed codecs using bits-per-dimension (bpd).
• Conduct ablations to isolate contributions of each architectural change.

Experimental results

Research questions

• RQ1 Can integer discrete flows (IDF) factorize all distributions via embedding into a larger class space?
• RQ2 How does gradient bias from straight-through estimators affect training efficiency and final performance of IDFs?
• RQ3 Do architectural changes (DenseNet++ blocks, normalization, invert-per-layer channel permutations) improve lossless compression performance of IDFs?
• RQ4 How does IDF++ compare with prior methods (IDF, hand-designed codecs) on standard compression benchmarks?

Key findings

• Embedding discrete data into Zd with infinite classes allows IDFs to factorize distributions, countering claims of limited flexibility.
• Straight-through gradient bias remains not as detrimental as previously thought; alternative estimators do not consistently improve results.
• Architecture choice heavily influences IDF depth/degradation; DenseNet++ and other modifications yield better performance with fewer flows.
• IDF++ with 4 flows per level matches or exceeds IDF with 8 flows per level, reducing parameters and compute by about half.
• On CIFAR-10, IDF++ with 4 flows per level achieves comparable or better bpd than IDF with 8 flows per level; on ImageNet variants, IDF++ matches or surpasses baselines.
• Compared to hand-designed codecs and prior deep density estimators, IDF++ shows competitive or superior compression performance, with quantitative gains reported in Table 1.