[Paper Review] When Does Label Smoothing Help?

Rafael Rios Müller, Simon Kornblith+1 more|arXiv (Cornell University)|Jun 6, 2019

Time Series Analysis and Forecasting17 references884 citations

TL;DR

The paper analyzes how label smoothing affects generalization, calibration, and knowledge distillation, showing it improves calibration and generalization but can hurt distillation due to information erasure in logits.

ABSTRACT

The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels. Smoothing the labels in this way prevents the network from becoming over-confident and label smoothing has been used in many state-of-the-art models, including image classification, language translation and speech recognition. Despite its widespread use, label smoothing is still poorly understood. Here we show empirically that in addition to improving generalization, label smoothing improves model calibration which can significantly improve beam-search. However, we also observe that if a teacher network is trained with label smoothing, knowledge distillation into a student network is much less effective. To explain these observations, we visualize how label smoothing changes the representations learned by the penultimate layer of the network. We show that label smoothing encourages the representations of training examples from the same class to group in tight clusters. This results in loss of information in the logits about resemblances between instances of different classes, which is necessary for distillation, but does not hurt generalization or calibration of the model's predictions.

Motivation & Objective

• Investigate why and when label smoothing improves neural network performance.
• Characterize how label smoothing changes penultimate-layer representations.
• Assess the impact of label smoothing on model calibration across tasks.
• Examine how label smoothing affects knowledge distillation and information transfer.

Proposed method

• Introduce a visualization method for penultimate-layer activations via projections.
• Quantify calibration using expected calibration error (ECE) and reliability diagrams.
• Evaluate calibration and accuracy across image classification and translation tasks with and without label smoothing.
• Analyze the effect of label smoothing on knowledge distillation using teacher–student setups.
• Estimate mutual information between inputs and logits to study information preservation under label smoothing.

Experimental results

Research questions

• RQ1 Does label smoothing improve model calibration and thus downstream tasks like beam-search?
• RQ2 How does label smoothing reshape penultimate-layer representations?
• RQ3 Why does label smoothing impair knowledge distillation despite improving teacher accuracy?
• RQ4 What is the relationship between label smoothing, mutual information, and information compression in networks?

Key findings

• Label smoothing improves calibration and can reduce over-confidence in predictions.
• Label smoothing leads to tighter, equally spaced clusters in penultimate-layer activations, indicating an information-erasing effect between classes.
• Label smoothing improves BLEU and calibration in translation tasks but yields worse NLL compared to hard targets.
• Distillation from teachers trained with label smoothing can perform worse than distillation from teachers trained with hard targets due to loss of logit information.
• Mutual information between inputs and logit differences decreases with label smoothing, indicating information erasure in representations.