[Paper Review] Early-Learning Regularization Prevents Memorization of Noisy Labels

Sheng Liu, Jonathan Niles‐Weed+2 more|arXiv (Cornell University)|Jun 30, 2020

Machine Learning and Data Classification53 references261 citations

TL;DR

The paper introduces Early-Learning Regularization (ELR) to prevent memorization of noisy labels by leveraging early-learning dynamics and a semi-supervised target-based regularization, achieving competitive or state-of-the-art results on benchmarks with noisy labels.

ABSTRACT

We propose a novel framework to perform classification via deep learning in the presence of noisy annotations. When trained on noisy labels, deep neural networks have been observed to first fit the training data with clean labels during an "early learning" phase, before eventually memorizing the examples with false labels. We prove that early learning and memorization are fundamental phenomena in high-dimensional classification tasks, even in simple linear models, and give a theoretical explanation in this setting. Motivated by these findings, we develop a new technique for noisy classification tasks, which exploits the progress of the early learning phase. In contrast with existing approaches, which use the model output during early learning to detect the examples with clean labels, and either ignore or attempt to correct the false labels, we take a different route and instead capitalize on early learning via regularization. There are two key elements to our approach. First, we leverage semi-supervised learning techniques to produce target probabilities based on the model outputs. Second, we design a regularization term that steers the model towards these targets, implicitly preventing memorization of the false labels. The resulting framework is shown to provide robustness to noisy annotations on several standard benchmarks and real-world datasets, where it achieves results comparable to the state of the art.

Motivation & Objective

• Motivate the study of training with noisy labels and identify early-learning and memorization as fundamental high-dimensional phenomena.
• Develop a regularization-based method that exploits early-learning to reduce memorization of incorrect labels.
• Leverage semi-supervised target estimation and regularization to steer gradients away from memorized wrong labels.
• Empirically evaluate ELR and ELR+ on standard noisy-label benchmarks and real-world datasets to benchmark against state-of-the-art methods.

Proposed method

• The authors analyze gradient dynamics of cross-entropy under noisy labels and identify how wrong-label gradients dominate later in training.
• They propose a regularization term L_ELR that adds lambda/n sum_i log(1 - <p_i, t_i>) to the cross-entropy, where p_i is the model output and t_i is a target probability estimate.
• Targets t_i are computed via temporal ensembling of past model outputs and can be enhanced by weight averaging and two-network setups (ELR+).
• They provide a gradient form showing ELR adds a correction g^i to the gradient, which boosts clean-label influence and dampens the wrong-label influence.
• Target probabilities t_i are updated with a momentum beta and can be derived from temporal ensembling or mean-weighted averages.
• They also discuss data augmentation (mixup) and network ensembles as components of ELR+ to improve robustness.

Experimental results

Research questions

• RQ1 Can early-learning dynamics be formalized as a fundamental phenomenon in high-dimensional classification with noisy labels?
• RQ2 Can a regularization approach, aligned with early-learning dynamics, prevent memorization of noisy labels without relying on sample selection?
• RQ3 How can target probabilities derived from model outputs be incorporated into the gradient to bias learning toward clean-label examples?
• RQ4 Do ELR and its enhanced variant ELR+ achieve competitive performance on CIFAR-10/100, Clothing1M, and WebVision compared to state-of-the-art noisy-label methods?

Key findings

• ELR consistently improves robustness to symmetric and asymmetric label noise on CIFAR-10/100 over several baselines.
• ELR+ further improves performance by combining temporal ensembling, weight averaging, two networks, and mixup data augmentation, achieving state-of-the-art performance on Clothing1M and competitive results on WebVision.
• Theoretical analysis shows early learning dominates gradients for clean labels initially, but wrong-label gradients can dominate later; ELR’s gradient correction counteracts memorization.
• Ablation studies indicate each component (temporal ensembling, weight averaging, two networks, mixup) independently contributes to gains, especially at higher noise levels.