[Paper Review] Unsupervised Label Noise Modeling and Loss Correction

Eric Arazo, Diego Ortego+3 more|arXiv (Cornell University)|Apr 25, 2019

Machine Learning and Data Classification198 citations

TL;DR

The paper models label noise with a two-component beta mixture on per-sample training loss to detect clean vs. noisy samples and applies dynamic bootstrapping loss, optionally combined with mixup, to robustly learn without requiring clean data.

ABSTRACT

Despite being robust to small amounts of label noise, convolutional neural networks trained with stochastic gradient methods have been shown to easily fit random labels. When there are a mixture of correct and mislabelled targets, networks tend to fit the former before the latter. This suggests using a suitable two-component mixture model as an unsupervised generative model of sample loss values during training to allow online estimation of the probability that a sample is mislabelled. Specifically, we propose a beta mixture to estimate this probability and correct the loss by relying on the network prediction (the so-called bootstrapping loss). We further adapt mixup augmentation to drive our approach a step further. Experiments on CIFAR-10/100 and TinyImageNet demonstrate a robustness to label noise that substantially outperforms recent state-of-the-art. Source code is available at https://git.io/fjsvE

Motivation & Objective

• Motivate robust learning when training data contains label noise without assuming access to clean data.
• Model clean/noisy sample ownership using an unsupervised loss-based beta mixture model.
• Develop a dynamic, per-sample loss correction (bootstrapping) informed by the noise model.
• Enhance robustness by integrating mixup data augmentation with the proposed loss correction.
• Demonstrate state-of-the-art performance on CIFAR-10/100 and TinyImageNet under varying noise levels.

Proposed method

• Model clean/noisy sample likelihoods by fitting a two-component beta mixture model (BMM) to training losses.
• Use EM to estimate beta parameters and posterior p(k|loss) for clean vs. noisy components.
• Derive a dynamic bootstrapping loss where per-sample weights w_i = p(k=1|loss_i) adjust the contribution of ground-truth labels vs. model predictions.
• Combine the dynamic bootstrapping with mixup data augmentation, creating a mixed loss that weights and augments both samples and labels using per-sample noise posteriors.
• Introduce a regularization term to prevent collapse to a single class during training.
• Provide a dynamic mixup variant that adapts the mixing strategy based on per-sample noise estimates to improve convergence under extreme label noise.

Experimental results

Research questions

• RQ1 Can an unsupervised, per-sample loss–based model separate clean from noisy labels without any clean subset?
• RQ2 Does a beta mixture model on sample losses provide reliable posterior probabilities to correct losses during training?
• RQ3 Can dynamic, per-sample bootstrapping improve robustness to label noise compared with static bootstrapping or standard cross-entropy?
• RQ4 Does coupling the proposed loss correction with mixup yield stronger robustness than mixup alone under high noise?

Key findings

• Beta mixture modeling of per-sample losses yields effective separation of clean vs. noisy samples, enabling per-sample loss correction.
• Dynamic bootstrapping (per-sample weighting) consistently outperforms static bootstrapping, especially at high noise levels (e.g., 80% and 90%).
• Hard bootstrapping with dynamic weights coupled with mixup (M-DYR-H) achieves substantial accuracy gains at high noise on CIFAR-10 (e.g., best 86.8% at 80% noise; 40.8% at 90%) and CIFAR-100 (best 12.5% at 80% noise; - at 90% in table).
• Joint dynamic bootstrapping and mixup (M-DYR-H/SH) significantly improves robustness over mixup alone, setting new baselines at high noise on CIFAR-10/100.
• The approach generalizes beyond CIFAR to TinyImageNet (MD-DYR-SH consistently outperforms baseline mixup).
• On Clothing1M, finetuning pre-trained networks limited unsupervised noise modeling, indicating some dataset-specific challenges.