[Paper Review] Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study

Zhiqiang Shen, Zechun Liu+4 more|arXiv (Cornell University)|Apr 1, 2021

Machine Learning and Data Classification40 references26 citations

TL;DR

The paper argues that label smoothing is not incompatible with knowledge distillation, showing that smoothing can enhance distillation in many settings but may fail under long-tailed or many-class regimes; it introduces a stability metric and analyzes effects across vision, BNNs, and NMT.

ABSTRACT

This work aims to empirically clarify a recently discovered perspective that label smoothing is incompatible with knowledge distillation. We begin by introducing the motivation behind on how this incompatibility is raised, i.e., label smoothing erases relative information between teacher logits. We provide a novel connection on how label smoothing affects distributions of semantically similar and dissimilar classes. Then we propose a metric to quantitatively measure the degree of erased information in sample's representation. After that, we study its one-sidedness and imperfection of the incompatibility view through massive analyses, visualizations and comprehensive experiments on Image Classification, Binary Networks, and Neural Machine Translation. Finally, we broadly discuss several circumstances wherein label smoothing will indeed lose its effectiveness. Project page: http://zhiqiangshen.com/projects/LS_and_KD/index.html.

Motivation & Objective

• Clarify the debated incompatibility between label smoothing and knowledge distillation.
• Explain how label smoothing affects intra-class and inter-class representations in teacher models.
• Propose a quantitative stability metric to measure erased information from label smoothing.
• Empirically evaluate KD with label smoothing across ImageNet, CUB200-2011, iMaterialist, BNNs, and NMT.
• Identify conditions under which label smoothing is less effective in KD (long-tail, many classes).

Proposed method

• Model training and visualization to compare teachers trained with hard vs. smoothed labels.
• Definition and use of a stability metric measuring intra-class probability variance.
• Knowledge distillation setup using soft labels from teachers without hard labels (KL divergence between teacher and student outputs).
• Extensive experiments across ImageNet-1K, CUB200-2011, iMaterialist, binary networks, and German→English NMT.
• Ablations with different teacher accuracies and architectures to assess supervision quality for KD.

Experimental results

Research questions

• RQ1 Does label smoothing in teacher networks suppress the effectiveness of knowledge distillation?
• RQ2 How does label smoothing affect intra-class representations and the transfer of knowledge to students?
• RQ3 Can a stability metric quantify the erased information and predict KD success?
• RQ4 Under what data regimes (long-tailed or many classes) does label smoothing become less effective for KD?
• RQ5 What factors determine a better teacher for knowledge distillation when using label smoothing?

Key findings

• Label smoothing does not impair KD performance; smoothed teachers can yield comparable or better validation accuracy despite higher training loss.
• Higher accuracy teachers generally distill stronger students, though the relationship is not strictly linear and depends on the student capacity.
• Label smoothing reduces intra-class variance (stability) and flattens predictions, yet preserves inter-class relation structure, aiding KD on similar classes.
• The proposed stability metric correlates with model accuracy and can serve as an indicator of teacher quality for KD.
• Long-tailed distributions and a larger number of classes reduce the effectiveness of label smoothing in KD; results also vary with dataset complexity and class count.