[Paper Review] Model compression via distillation and quantization

Antonio Polino, Razvan Pascanu+1 more|arXiv (Cornell University)|Feb 15, 2018

Advanced Neural Network Applications26 references262 citations

TL;DR

The paper introduces two methods—quantized distillation and differentiable quantization—to compress deep networks by distilling from a full‑precision teacher into shallower, quantized students, with strong accuracy retention and significant compression across vision and language tasks.

ABSTRACT

Deep neural networks (DNNs) continue to make significant advances, solving tasks from image classification to translation or reinforcement learning. One aspect of the field receiving considerable attention is efficiently executing deep models in resource-constrained environments, such as mobile or embedded devices. This paper focuses on this problem, and proposes two new compression methods, which jointly leverage weight quantization and distillation of larger teacher networks into smaller student networks. The first method we propose is called quantized distillation and leverages distillation during the training process, by incorporating distillation loss, expressed with respect to the teacher, into the training of a student network whose weights are quantized to a limited set of levels. The second method, differentiable quantization, optimizes the location of quantization points through stochastic gradient descent, to better fit the behavior of the teacher model. We validate both methods through experiments on convolutional and recurrent architectures. We show that quantized shallow students can reach similar accuracy levels to full-precision teacher models, while providing order of magnitude compression, and inference speedup that is linear in the depth reduction. In sum, our results enable DNNs for resource-constrained environments to leverage architecture and accuracy advances developed on more powerful devices.

Motivation & Objective

• Leverage high-accuracy full-precision teachers to improve compressed student models.
• Combine distillation with weight quantization to achieve simultaneous depth and width reductions.
• Validate methods across CNN, RNN, and translation tasks to demonstrate generality and practical gains.
• Quantify compression and speedups while preserving accuracy on standard benchmarks.

Proposed method

• Define weight quantization with scaling, bucketing, and both uniform/non-uniform schemes.
• Introduce quantized distillation where the student is trained with distillation loss using quantized weights.
• Develop differentiable quantization that learns quantization points p via SGD by backpropagating through the quantization function.
• Apply the methods to CNNs (e.g., ResNet variants), Wide ResNets, LSTMs in OpenNMT, and WMT translation setups.
• Analyze compression benefits, storage, and inference speedups, including bucketed and Huffman-encoded representations.

Experimental results

Research questions

• RQ1 Can distillation coupled with quantization yield high-accuracy, compressed models suitable for resource-constrained environments?
• RQ2 How do quantized distillation and differentiable quantization compare in accuracy, convergence, and efficiency across vision and language tasks?
• RQ3 What is the impact of bit-width, bucket size, and architecture on compression-accuracy trade-offs?
• RQ4 Does distillation loss outperform standard loss when training quantized models?
• RQ5 How do these methods scale to large datasets and architectures (e.g., ImageNet, WMT)?

Key findings

• Quantized shallow students can approach full-precision teacher accuracy while achieving up to an order of magnitude in compression.
• Quantized distillation often outperforms post-mortem quantization and differentiable quantization on 2- and 4-bit settings across tasks.
• In ImageNet, a 4-bit quantized, distilled 2xResNet18 can reach accuracy comparable to a ResNet34 teacher while being smaller and faster.
• On CIFAR-10, differentiable quantization and quantized distillation yield near-teacher accuracy at 4 bits, with larger gains using distillation loss.
• OpenNMT and WMT experiments show distillation helps maintain BLEU and perplexity close to teacher levels at reduced sizes.