[论文解读] Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations

Eirikur Agustsson, Fabian Mentzer+5 more|arXiv (Cornell University)|Apr 3, 2017

Advanced Image Processing Techniques参考文献 37被引用 262

一句话总结

论文介绍了一种端到端训练的软到硬向量量化框架，用于学习可压缩的特征表征和模型，在图像和DNN压缩方面取得具有竞争力的结果。

ABSTRACT

We present a new approach to learn compressible representations in deep architectures with an end-to-end training strategy. Our method is based on a soft (continuous) relaxation of quantization and entropy, which we anneal to their discrete counterparts throughout training. We showcase this method for two challenging applications: Image compression and neural network compression. While these tasks have typically been approached with different methods, our soft-to-hard quantization approach gives results competitive with the state-of-the-art for both.

研究动机与目标

• Motivate learning compressible representations for deep networks and data types.
• Jointly optimize model parameters, quantization levels, and entropy of the symbol stream.
• Provide a unified framework for compression of both features in networks and whole models.

提出的方法

• Introduce a soft (continuous) relaxation of quantization and entropy with an annealing schedule from soft to hard assignments.
• Model encoder E as selecting nearest centers from a learned codebook; decoder D reconstructs from symbol indices.
• Estimate entropy via a differentiable soft histogram and a soft entropy loss that upper-bounds the true entropy.
• Anneal the soft quantization to hard assignments to enable end-to-end differentiable training.
• Apply vector quantization (not just scalar) to better capture bottleneck statistics.
• Demonstrate end-to-end learning for both image compression via a compressive autoencoder and DNN model compression (ResNet on CIFAR-10).

实验结果

研究问题

• RQ1 Can soft-to-hard vector quantization be trained end-to-end to minimize distortion plus rate (D + βR) in deep networks?
• RQ2 Does learning the quantization levels jointly with weights improve compressibility for both image data and model parameters?
• RQ3 How does vector quantization compare to scalar quantization in learned compression scenarios?
• RQ4 Can histogram-based entropy estimation without strong parametric assumptions yield competitive results?

主要发现

Method	Acc	Comp.
Original model	92.6	1.00
Pruning + ft. + index coding + H. Coding [12]	92.6	4.52
Pruning + ft. + k-means + ft. + I.C. + H.C. [11]	92.6	18.25
Pruning + ft. + Hessian-weighted k-means + ft. + I.C. + H.C.	92.7	20.51
Pruning + ft. + Uniform quantization + ft. + I.C. + H.C.	92.7	22.17
Pruning + ft. + Iterative ECSQ + ft. + I.C. + H.C.	92.7	21.01
Soft-to-Hard Annealing + ft. + H. Coding ( ours)	92.1	19.15
Soft-to-Hard Annealing + ft. + A. Coding ( ours)	92.1	20.15

• Achieves competitive performance with state-of-the-art methods for both image compression and DNN model compression.
• Vector quantization with soft-to-hard annealing improves rate-distortion trade-offs over scalar quantization.
• Entropy loss based on soft histograms provides differentiable guidance for compressibility.
• On CIFAR-10 with a 32-layer ResNet, achieves ~19–20× compression with minimal loss in accuracy.
• For image compression, SHA outperforms JPEG/JPEG 2000 at high compression rates and is competitive with BPG on several datasets.