[Paper Review] Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy

Asit Mishra, Debbie Marr|arXiv (Cornell University)|Nov 15, 2017

Sensor Technology and Measurement SystemsComputer Science157 citations

TL;DR

This paper shows how knowledge distillation can significantly boost accuracy of low-precision DNNs, achieving state-of-the-art results for ternary and 4-bit ResNet on ImageNet through three distillation schemes.

ABSTRACT

Deep learning networks have achieved state-of-the-art accuracies on computer vision workloads like image classification and object detection. The performant systems, however, typically involve big models with numerous parameters. Once trained, a challenging aspect for such top performing models is deployment on resource constrained inference systems - the models (often deep networks or wide networks or both) are compute and memory intensive. Low-precision numerics and model compression using knowledge distillation are popular techniques to lower both the compute requirements and memory footprint of these deployed models. In this paper, we study the combination of these two techniques and show that the performance of low-precision networks can be significantly improved by using knowledge distillation techniques. Our approach, Apprentice, achieves state-of-the-art accuracies using ternary precision and 4-bit precision for variants of ResNet architecture on ImageNet dataset. We present three schemes using which one can apply knowledge distillation techniques to various stages of the train-and-deploy pipeline.

Motivation & Objective

• Demonstrate how combining quantization with knowledge distillation improves low-precision DNN accuracy.
• Quantify gains of distillation across ResNet-18/34/50/101 on ImageNet.
• Present three practical schemes to apply distillation during training and deployment of low-precision networks.
• Compare against prior low-precision methods and establish new state-of-the-art results for sub-8-bit networks.

Proposed method

• Define a teacher-student (apprentice) framework where the teacher is full-precision and the apprentice is low-precision.
• Quantize weights (ternary or 4-bit) and activations (8-bit/4-bit) and preserve first/last layers.
• Propose three schemes: A) joint training of teacher and apprentice; B) apprentice trained with fixed teacher logits; C) fine-tune a pre-trained full-precision apprentice after lowering precision.
• Use a loss combining ground-truth, teacher logits, and apprentice logits with calibrated weights (α=1, β=0.5, γ=0.5).
• Evaluate on ImageNet with ResNet backbones (18, 34, 50, 101) under various precision configurations.
• Compare against TTQ and WRPN baselines and report improvements.
• Discuss hyper-parameter choices and observed saturation effects on teacher-student guidance.

Experimental results

Research questions

• RQ1 Can knowledge distillation substantially recover or surpass the accuracy of low-precision networks on ImageNet?
• RQ2 How do three distinct distillation schemes compare in improving ternary and 4-bit ResNet models?
• RQ3 What is the impact of teacher capacity and target precision on final apprentice performance?
• RQ4 Do the gains hold across different ResNet depths (18, 34, 50) and across training vs. fine-tuning setups?

Key findings

• Three distillation schemes yield state-of-the-art accuracy for ternary and 4-bit weights on ResNet variants.
• Scheme-A (joint training) often provides the strongest gains, with full-precision teachers guiding low-precision students.
• Scheme-B accelerates convergence (fewer epochs) while reaching comparable accuracies.
• Scheme-C (fine-tuning a pre-trained full-precision model after lowering precision) yields marginally better results in some configurations, e.g., ResNet-50 with ternary weights.
• Across tested configurations, Apprentice reduces the gap to full-precision accuracy by significant margins and improves over prior low-precision baselines (e.g., TTQ, Mellempudi et al.).
• Ternary networks achieve competitive model sizes with substantial accuracy recovery, approaching within about 1% of full-precision accuracy in several cases.