[Paper Review] Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective

Helong Zhou, Liangchen Song+5 more|arXiv (Cornell University)|Feb 1, 2021

Machine Learning and Algorithms40 references44 citations

TL;DR

The paper analyzes how soft labels in knowledge distillation induce a sample-wise bias-variance tradeoff and introduces weighted soft labels to adaptively balance this tradeoff, validated by experiments on standard benchmarks.

ABSTRACT

Knowledge distillation is an effective approach to leverage a well-trained network or an ensemble of them, named as the teacher, to guide the training of a student network. The outputs from the teacher network are used as soft labels for supervising the training of a new network. Recent studies \citep{muller2019does,yuan2020revisiting} revealed an intriguing property of the soft labels that making labels soft serves as a good regularization to the student network. From the perspective of statistical learning, regularization aims to reduce the variance, however how bias and variance change is not clear for training with soft labels. In this paper, we investigate the bias-variance tradeoff brought by distillation with soft labels. Specifically, we observe that during training the bias-variance tradeoff varies sample-wisely. Further, under the same distillation temperature setting, we observe that the distillation performance is negatively associated with the number of some specific samples, which are named as regularization samples since these samples lead to bias increasing and variance decreasing. Nevertheless, we empirically find that completely filtering out regularization samples also deteriorates distillation performance. Our discoveries inspired us to propose the novel weighted soft labels to help the network adaptively handle the sample-wise bias-variance tradeoff. Experiments on standard evaluation benchmarks validate the effectiveness of our method. Our code is available at \url{https://github.com/bellymonster/Weighted-Soft-Label-Distillation}.

Motivation & Objective

• Motivate analysis of soft labels in KD from a bias-variance perspective.
• Characterize how bias and variance evolve during KD training on a per-sample basis.
• Identify regularization samples that disproportionately affect KD performance.
• Propose and validate weighted soft labels to adaptively manage sample-wise bias-variance during training.

Proposed method

• Decompose KD loss into bias-variance components using KL divergence-based analysis.
• Compare bias-variance decomposition of direct training (cross-entropy) versus distillation loss (KD).
• Show that regularization samples exist where variance reduction dominates and bias increases.
• Introduce a temperature-insensitive weighting scheme for soft labels based on teacher and student predictions (weighted soft labels).
• Combine L_ce with the weighted KD loss (L_wsl) for training, with a balancing hyperparameter α.

Experimental results

Research questions

• RQ1 How do bias and variance evolve during training when using soft labels in knowledge distillation?
• RQ2 What is the role of regularization samples in KD performance under fixed distillation temperature?
• RQ3 Can a sample-wise weighting scheme mitigate negative effects of regularization samples and improve KD performance?

Key findings

• Soft labels act as both supervisory signals and regularizers, leading to a sample-wise bias-variance tradeoff.
• Under the same temperature, a subset of samples (regularization samples) negatively correlates with KD performance due to increased bias and decreased variance benefit.
• Completely filtering out regularization samples hurts performance, indicating they contain exploitable information for KD.
• A simple weighted soft-label scheme (L_wsl) mitigates the adverse impact of regularization samples and improves KD performance.
• Experiments on CIFAR-100 and ImageNet show competitive or superior results to state-of-the-art KD methods across various teacher-student pairs.