[Paper Review] Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations

Eirikur Agustsson, Fabian Mentzer+5 more|arXiv (Cornell University)|Apr 3, 2017

Advanced Image Processing Techniques37 references262 citations

TL;DR

The paper introduces a soft-to-hard vector quantization framework with end-to-end training to learn compressible feature representations and models, achieving competitive results for image and DNN compression.

ABSTRACT

We present a new approach to learn compressible representations in deep architectures with an end-to-end training strategy. Our method is based on a soft (continuous) relaxation of quantization and entropy, which we anneal to their discrete counterparts throughout training. We showcase this method for two challenging applications: Image compression and neural network compression. While these tasks have typically been approached with different methods, our soft-to-hard quantization approach gives results competitive with the state-of-the-art for both.

Motivation & Objective

• Motivate learning compressible representations for deep networks and data types.
• Jointly optimize model parameters, quantization levels, and entropy of the symbol stream.
• Provide a unified framework for compression of both features in networks and whole models.

Proposed method

• Introduce a soft (continuous) relaxation of quantization and entropy with an annealing schedule from soft to hard assignments.
• Model encoder E as selecting nearest centers from a learned codebook; decoder D reconstructs from symbol indices.
• Estimate entropy via a differentiable soft histogram and a soft entropy loss that upper-bounds the true entropy.
• Anneal the soft quantization to hard assignments to enable end-to-end differentiable training.
• Apply vector quantization (not just scalar) to better capture bottleneck statistics.
• Demonstrate end-to-end learning for both image compression via a compressive autoencoder and DNN model compression (ResNet on CIFAR-10).

Experimental results

Research questions

• RQ1 Can soft-to-hard vector quantization be trained end-to-end to minimize distortion plus rate (D + βR) in deep networks?
• RQ2 Does learning the quantization levels jointly with weights improve compressibility for both image data and model parameters?
• RQ3 How does vector quantization compare to scalar quantization in learned compression scenarios?
• RQ4 Can histogram-based entropy estimation without strong parametric assumptions yield competitive results?

Key findings

• Achieves competitive performance with state-of-the-art methods for both image compression and DNN model compression.
• Vector quantization with soft-to-hard annealing improves rate-distortion trade-offs over scalar quantization.
• Entropy loss based on soft histograms provides differentiable guidance for compressibility.
• On CIFAR-10 with a 32-layer ResNet, achieves ~19–20× compression with minimal loss in accuracy.
• For image compression, SHA outperforms JPEG/JPEG 2000 at high compression rates and is competitive with BPG on several datasets.