[Paper Review] Knowledge Distillation Based on Transformed Teacher Matching

Kaixiang Zheng, En‐hui Yang|arXiv (Cornell University)|Feb 17, 2024

Educational Technology and Assessment4 citations

TL;DR

This paper proposes Transformed Teacher Matching (TTM), a knowledge distillation method that removes temperature scaling from the student network, introducing an inherent Rényi entropy regularizer that improves generalization. It further introduces Weighted TTM (WTTM), a sample-adaptive variant that enhances matching accuracy, achieving state-of-the-art performance on ImageNet with 72.19% top-1 accuracy for ResNet-18 distilled from ResNet-34.

ABSTRACT

As a technique to bridge logit matching and probability distribution matching, temperature scaling plays a pivotal role in knowledge distillation (KD). Conventionally, temperature scaling is applied to both teacher's logits and student's logits in KD. Motivated by some recent works, in this paper, we drop instead temperature scaling on the student side, and systematically study the resulting variant of KD, dubbed transformed teacher matching (TTM). By reinterpreting temperature scaling as a power transform of probability distribution, we show that in comparison with the original KD, TTM has an inherent Rényi entropy term in its objective function, which serves as an extra regularization term. Extensive experiment results demonstrate that thanks to this inherent regularization, TTM leads to trained students with better generalization than the original KD. To further enhance student's capability to match teacher's power transformed probability distribution, we introduce a sample-adaptive weighting coefficient into TTM, yielding a novel distillation approach dubbed weighted TTM (WTTM). It is shown, by comprehensive experiments, that although WTTM is simple, it is effective, improves upon TTM, and achieves state-of-the-art accuracy performance. Our source code is available at https://github.com/zkxufo/TTM.

Motivation & Objective

• To investigate whether temperature scaling should be applied to both teacher and student in knowledge distillation.
• To understand the theoretical and empirical impact of removing temperature scaling from the student network.
• To develop a more effective distillation method that enhances generalization and matching accuracy through inherent regularization.
• To propose a sample-adaptive weighting mechanism that improves upon TTM for better teacher-student probability distribution matching.

Proposed method

• TTM removes temperature scaling from the student logits, reformulating knowledge distillation to minimize cross-entropy loss and KL divergence between the temperature-scaled teacher output and unnormalized student output.
• The method reinterprets temperature scaling as a power transform of probability distributions, revealing that TTM inherently includes a Rényi entropy regularizer in its objective function.
• WTTM introduces a sample-adaptive weighting coefficient $ U_{1/T}(p^t) $ that dynamically adjusts the influence of the teacher’s probability distribution based on its confidence and temperature.
• The balancing weight $ \beta $ in TTM is set to maintain the same relative loss ratio as in standard KD, ensuring fair comparison.
• Theoretical analysis shows that removing temperature scaling from the student introduces a regularization effect that improves generalization.
• WTTM maintains computational complexity similar to standard KD while significantly improving performance through adaptive matching.

Experimental results

Research questions

• RQ1 Does applying temperature scaling only to the teacher, and not to the student, lead to better generalization in knowledge distillation?
• RQ2 What is the theoretical mechanism behind the improved performance of this variant, and how does it differ from standard KD?
• RQ3 Can a sample-adaptive weighting strategy further enhance the matching between the student and the transformed teacher distribution?
• RQ4 How does the inherent Rényi entropy regularizer in TTM affect the smoothness and generalization of student predictions?

Key findings

• TTM achieves better generalization than standard KD due to an inherent Rényi entropy regularizer introduced by removing temperature scaling from the student.
• Students trained with TTM exhibit significantly higher average Shannon entropy in their output distributions, indicating smoother and more robust predictions.
• WTTM outperforms both KD and TTM, achieving 72.19% top-1 accuracy on ImageNet when distilling ResNet-18 from ResNet-34, surpassing most complex feature-based distillation methods.
• Even without the cross-entropy loss, WTTM outperforms KD with the same loss, demonstrating its robustness and strong inductive bias.
• As the teacher model improves, the student’s performance under WTTM increases consistently and surpasses other methods, indicating strong scalability.
• The average KL divergence $ D(p^t_T || q) $ decreases faster and to a lower value in WTTM than in TTM, confirming more accurate transformed teacher matching.