[Paper Review] Self-Distillation as Instance-Specific Label Smoothing

Zhilu Zhang, Mert R. Sabuncu|arXiv (Cornell University)|Jun 9, 2020

Machine Learning and Data ClassificationComputer Science41 references52 citations

TL;DR

The paper interprets self-distillation as instance-specific regularization within a MAP framework, linking distillation to label smoothing and introducing Beta smoothing to promote confidence diversity without a separate teacher.

ABSTRACT

It has been recently demonstrated that multi-generational self-distillation can improve generalization. Despite this intriguing observation, reasons for the enhancement remain poorly understood. In this paper, we first demonstrate experimentally that the improved performance of multi-generational self-distillation is in part associated with the increasing diversity in teacher predictions. With this in mind, we offer a new interpretation for teacher-student training as amortized MAP estimation, such that teacher predictions enable instance-specific regularization. Our framework allows us to theoretically relate self-distillation to label smoothing, a commonly used technique that regularizes predictive uncertainty, and suggests the importance of predictive diversity in addition to predictive uncertainty. We present experimental results using multiple datasets and neural network architectures that, overall, demonstrate the utility of predictive diversity. Finally, we propose a novel instance-specific label smoothing technique that promotes predictive diversity without the need for a separately trained teacher model. We provide an empirical evaluation of the proposed method, which, we find, often outperforms classical label smoothing.

Motivation & Objective

• Investigate why multi-generational self-distillation improves generalization.
• Provide a MAP-based interpretation of teacher-student training.
• Relate distillation to label smoothing and emphasize the role of predictive diversity.
• Propose Beta smoothing as an efficient instance-specific regularization technique.
• Demonstrate improved calibration through regularization on the probability simplex.

Proposed method

• Model the distillation process as amortized MAP estimation of softmax outputs.
• Relate teacher predictions to instance-specific priors on the output distribution.
• Compare self-distillation with classical label smoothing through systematic experiments.
• Introduce Beta smoothing to implement instance-specific prior without a separate teacher.
• Analyze predictive uncertainty and confidence diversity using entropy-based metrics.
• Evaluate calibration improvements via expected calibration error (ECE) across datasets.

Experimental results

Research questions

• RQ1 Does increasing diversity in teacher predictions correlate with improved student performance in self-distillation?
• RQ2 Can self-distillation be theoretically connected to label smoothing through a MAP framework?
• RQ3 Does instance-specific regularization (including Beta smoothing) outperform traditional label smoothing?
• RQ4 Can Beta smoothing offer calibration benefits comparable to or better than self-distillation?
• RQ5 What is the role of predictive diversity in improving generalization and calibration?

Key findings

• Sequential self-distillation shows improved test accuracy and better calibration across generations.
• Higher diversity in teacher predictions is associated with better student performance.
• Label smoothing increases predictive uncertainty but may not achieve diversity; instance-specific priors help.
• Beta smoothing often outperforms classical label smoothing and can match self-distillation without a separate teacher.
• The MAP perspective explains distillation as a form of instance-specific regularization, improving calibration.
• Temperature-adjusted teacher predictions can greatly boost student accuracy by controlling uncertainty and diversity.