[Paper Review] Does Knowledge Distillation Really Work?

Samuel Stanton, Pavel Izmailov+3 more|arXiv (Cornell University)|Jun 10, 2021

Domain Adaptation and Few-Shot Learning43 references11 citations

TL;DR

This paper investigates whether knowledge distillation truly transfers knowledge from large teacher models to smaller student networks, finding that despite improvements in student generalization, fidelity—measured by predictive distribution alignment—remains surprisingly low. The core issue is optimization difficulty, not capacity or data quality, as students often fail to converge to optimal solutions even when initialized near the teacher's weights.

ABSTRACT

Knowledge distillation is a popular technique for training a small student network to emulate a larger teacher model, such as an ensemble of networks. We show that while knowledge distillation can improve student generalization, it does not typically work as it is commonly understood: there often remains a surprisingly large discrepancy between the predictive distributions of the teacher and the student, even in cases when the student has the capacity to perfectly match the teacher. We identify difficulties in optimization as a key reason for why the student is unable to match the teacher. We also show how the details of the dataset used for distillation play a role in how closely the student matches the teacher -- and that more closely matching the teacher paradoxically does not always lead to better student generalization.

Motivation & Objective

• To investigate why knowledge distillation frequently fails to achieve high predictive fidelity between teacher and student models despite strong generalization gains.
• To determine whether the discrepancy in predictions between teacher and student models stems from optimization difficulties, data quality, or identifiability issues.
• To decouple the concepts of distillation fidelity and generalization, showing that high accuracy does not imply high fidelity.
• To evaluate whether interventions like data augmentation, extended training, and shared initialization improve fidelity, and if so, under what conditions.

Proposed method

• Conducted experiments on CIFAR-100 using ResNet-20 and ResNet-56 architectures for both self-distillation and ensemble distillation.
• Systematically evaluated the impact of distillation dataset size by augmenting real data with GAN-generated samples to improve data quality and coverage.
• Compared student performance under different initialization schemes, including random initialization and initialization using the teacher’s initial weights.
• Measured fidelity using agreement rate, KL divergence between predictive distributions, and centered kernel alignment (CKA) of network preactivations.
• Evaluated the effect of training duration, data augmentation (e.g., MixUp), and optimizer choice (e.g., Adam) on fidelity and generalization.
• Used multiple random seeds (10 trials) to estimate mean and standard deviation for all metrics, ensuring robustness of results.

Experimental results

Research questions

• RQ1 To what extent does knowledge distillation achieve high predictive fidelity between teacher and student models in modern deep learning setups?
• RQ2 Does improving the quality or size of the distillation dataset lead to better student fidelity, even when the student has sufficient capacity?
• RQ3 Can shared weight initialization between teacher and student significantly improve distillation fidelity, or is the problem fundamentally optimization-based?
• RQ4 Is the failure to match the teacher’s predictions due to identifiability issues or due to suboptimal convergence during training?
• RQ5 What is the relationship between student generalization (accuracy) and distillation fidelity (distribution alignment), and do they always improve together?

Key findings

• Even with sufficient model capacity, knowledge distillation often fails to achieve high fidelity, as evidenced by a large discrepancy in predictive distributions between teacher and student models.
• In self-distillation, increasing the distillation dataset size with GAN-generated samples improved fidelity but reduced test accuracy, indicating a trade-off between optimization complexity and data quality.
• When the student is initialized near the teacher’s final weights, it converges to the same loss basin and achieves nearly 100% agreement, demonstrating that the optimization problem is the root cause of poor fidelity.
• Shared initialization between teacher and student increases CKA (from 77.174 to 77.098) but does not meaningfully improve functional agreement or predictive KL, indicating that initialization alone is insufficient.
• Despite interventions like MixUp, extended training, and careful data augmentation, the student still fails to match the teacher’s predictions, confirming that optimization dynamics are the primary obstacle.
• High student accuracy does not imply high fidelity; in fact, the most accurate models are not always the most faithful to the teacher’s predictive distribution.