[Paper Review] Knowledge Distillation from A Stronger Teacher

Tao Huang, Shan You+3 more|arXiv (Cornell University)|May 21, 2022

Advanced Neural Network Applications95 citations

TL;DR

DIST introduces a correlation-based relational distillation that relaxes exact output matching when the teacher is stronger, achieving consistent state-of-the-art gains across image classification, object detection, and semantic segmentation.

ABSTRACT

Unlike existing knowledge distillation methods focus on the baseline settings, where the teacher models and training strategies are not that strong and competing as state-of-the-art approaches, this paper presents a method dubbed DIST to distill better from a stronger teacher. We empirically find that the discrepancy of predictions between the student and a stronger teacher may tend to be fairly severer. As a result, the exact match of predictions in KL divergence would disturb the training and make existing methods perform poorly. In this paper, we show that simply preserving the relations between the predictions of teacher and student would suffice, and propose a correlation-based loss to capture the intrinsic inter-class relations from the teacher explicitly. Besides, considering that different instances have different semantic similarities to each class, we also extend this relational match to the intra-class level. Our method is simple yet practical, and extensive experiments demonstrate that it adapts well to various architectures, model sizes and training strategies, and can achieve state-of-the-art performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at: https://github.com/hunto/DIST_KD .

Motivation & Objective

• Investigate why stronger teachers can destabilize vanilla KD and how to address it.
• Propose a relation-based distillation loss that preserves teacher–student relations rather than exact outputs.
• Extend rel distillation to intra-class relations to capture instance-level similarities.
• Demonstrate effectiveness across diverse architectures, strategies, and vision tasks (classification, detection, segmentation).

Proposed method

• Replace KL-divergence matching with a relation-based loss that preserves inter-class relations via Pearson correlation.
• Define inter-class relation loss L_inter as the mean Pearson-distance between teacher and student prediction vectors across a batch.
• Define intra-class relation loss L_intra as the mean Pearson-distance across classes by comparing teacher and student prediction matrices column-wise.
• Optimize a joint training objective L_tr = alpha * L_cls + beta * L_inter + gamma * L_intra.
• Show that matching relative relations rather than exact probabilities yields robust distillation under stronger teachers and training strategies.

Experimental results

Research questions

• RQ1 How does a stronger teacher affect standard knowledge distillation performance?
• RQ2 Can preserving relations between teacher and student predictions (instead of exact values) improve KD when teachers are stronger?
• RQ3 What is the impact of inter-class and intra-class relational distillation separately and jointly?
• RQ4 Are the proposed DIST losses effective across image classification, object detection, and semantic segmentation with various architectures and training strategies?

Key findings

• DIST outperforms vanilla KD and several state-of-the-art KD methods on ImageNet across multiple teacher–student pairs (e.g., ResNet-18 student with ResNet-34 teacher: 72.07% Top-1 with DIST vs 69.76% with KD).
• With larger teachers, DIST yields larger gains (e.g., ResNet-18 with ResNet-34: +2.31 over baseline and +0.86 over KD; ResNet-18 with ResNet-152: +1.93 over KD).
• DIST achieves 82.3% Top-1 on Swin-T with a stronger Swin-L teacher, outperforming conventional KD by 1%.
• DIST consistently improves performance on CIFAR-100 and ImageNet under strong training strategies, and also delivers competitive results on COCO object detection and Cityscapes semantic segmentation.
• Ablation studies show both inter-class and intra-class relations contribute to gains, with the combination providing the best performance.
• Training with only the KD loss (no ground-truth) still yields improvements with DIST, indicating effective distillation of relational signals.