[Paper Review] Preparing Lessons: Improve Knowledge Distillation with Better Supervision

Tiancheng Wen, Shenqi Lai+1 more|arXiv (Cornell University)|Nov 18, 2019

Advanced Neural Network Applications51 references20 citations

TL;DR

This paper proposes Knowledge Adjustment (KA) and Dynamic Temperature Distillation (DTD) to improve knowledge distillation by addressing incorrect and overly uncertain teacher supervision. KA corrects misclassified teacher predictions using ground truth labels, while DTD uses sample-specific temperature scaling to reduce uncertainty in soft labels, leading to state-of-the-art performance on CIFAR-100, CINIC-10, and Tiny ImageNet when combined with other KD methods.

ABSTRACT

Knowledge distillation (KD) is widely used for training a compact model with the supervision of another large model, which could effectively improve the performance. Previous methods mainly focus on two aspects: 1) training the student to mimic representation space of the teacher; 2) training the model progressively or adding extra module like discriminator. Knowledge from teacher is useful, but it is still not exactly right compared with ground truth. Besides, overly uncertain supervision also influences the result. We introduce two novel approaches, Knowledge Adjustment (KA) and Dynamic Temperature Distillation (DTD), to penalize bad supervision and improve student model. Experiments on CIFAR-100, CINIC-10 and Tiny ImageNet show that our methods get encouraging performance compared with state-of-the-art methods. When combined with other KD-based methods, the performance will be further improved.

Motivation & Objective

• To address the problem of bad supervision in knowledge distillation, where teacher models make incorrect or overly uncertain predictions.
• To improve student model performance by refining the quality of supervision from teacher networks.
• To reduce genetic errors and enhance model discrimination through better-structured soft labels.
• To develop methods that are universally applicable and compatible with existing KD frameworks.
• To investigate whether correcting teacher errors and reducing uncertainty can lead to consistent performance gains across diverse datasets.

Proposed method

• Knowledge Adjustment (KA) fixes teacher predictions on misclassified samples by replacing them with ground truth labels before loss computation.
• KA uses two implementations: one with a hard label correction and another with label regularization, both ensuring correct supervision.
• Dynamic Temperature Distillation (DTD) introduces sample-wise temperature scaling to reduce uncertainty in soft targets from the teacher.
• DTD dynamically adjusts the temperature per sample based on prediction confidence, promoting more discriminative soft labels.
• The method uses a differentiable formulation to compute soft targets with adaptive temperature, improving training stability.
• Both KA and DTD are compatible with standard KD and can be combined with other distillation techniques like AT and NST.

Experimental results

Research questions

• RQ1 How does incorrect supervision from teacher models affect student performance in knowledge distillation?
• RQ2 To what extent can correcting teacher mispredictions improve student generalization and reduce genetic errors?
• RQ3 Can dynamic temperature scaling reduce uncertainty in soft labels and improve student discrimination?
• RQ4 How do KA and DTD perform when combined with other KD-based methods?
• RQ5 Does the quality of supervision significantly impact distillation performance on heterogeneous teacher-student setups?

Key findings

• KA and DTD independently improve accuracy on CIFAR-100, CINIC-10, and Tiny ImageNet, with DTD-KA achieving state-of-the-art performance.
• On Tiny ImageNet, DTD-KA achieved the highest top-1 accuracy and reduced genetic errors more than other methods.
• Combining KA and DTD with existing methods like AT and NST led to further performance gains, indicating strong compatibility.
• The method reduced genetic errors significantly, especially on Tiny ImageNet, suggesting a higher proportion of misclassified teacher samples on this dataset.
• Experiments show that nearly half of the student’s errors are linked to teacher’s incorrect predictions, validating the need for supervision correction.
• DTD effectively mitigates uncertainty in soft labels by adapting temperature per sample, leading to more reliable supervision.