[Paper Review] SELF: Learning to Filter Noisy Labels with Self-Ensembling

Duc Tam Nguyen, Chaithanya Kumar Mummadi+4 more|arXiv (Cornell University)|Oct 4, 2019

Machine Learning and Data Classification25 references56 citations

TL;DR

SELF progressively filters out potentially noisy labels using a self-ensemble of predictions and Mean Teacher training, turning noisy supervision into a semi-supervised setting and boosting robustness to label noise across datasets.

ABSTRACT

Deep neural networks (DNNs) have been shown to over-fit a dataset when being trained with noisy labels for a long enough time. To overcome this problem, we present a simple and effective method self-ensemble label filtering (SELF) to progressively filter out the wrong labels during training. Our method improves the task performance by gradually allowing supervision only from the potentially non-noisy (clean) labels and stops learning on the filtered noisy labels. For the filtering, we form running averages of predictions over the entire training dataset using the network output at different training epochs. We show that these ensemble estimates yield more accurate identification of inconsistent predictions throughout training than the single estimates of the network at the most recent training epoch. While filtered samples are removed entirely from the supervised training loss, we dynamically leverage them via semi-supervised learning in the unsupervised loss. We demonstrate the positive effect of such an approach on various image classification tasks under both symmetric and asymmetric label noise and at different noise ratios. It substantially outperforms all previous works on noise-aware learning across different datasets and can be applied to a broad set of network architectures.

Motivation & Objective

• Motivate robust learning when training data contain noisy labels from crowd or web annotations.
• Propose a simple, effective mechanism to identify and drop likely incorrect labels during training.
• Leverage the remaining clean labels for supervised learning while using the full dataset in an unsupervised loss.
• Demonstrate robustness across multiple architectures and datasets (CIFAR-10, CIFAR-100, ImageNet).
• Show superiority over prior noise-robust methods under both symmetric and asymmetric label noise.

Proposed method

• Maintain a running mean-teacher (Mean Teacher) ensemble to provide a stable supervisory signal.
• Build a moving-average of sample predictions across training epochs to form a self-ensemble prediction for each sample.
• Filter labels by keeping only samples whose current label matches the argmax of the ensemble prediction over time; remaining noisy samples are removed from the supervised loss but used in an unsupervised loss.
• Iteratively train on the filtered label set, re-evaluating and refining the label set until no further improvement is observed.
• Treat the problem as a semi-supervised task by utilizing the entire dataset in an unsupervised loss while supervising only on the filtered, cleaner subset.
• Optionally use a second ensemble (Mean Teacher) to stabilize training and combine model and prediction ensembles for robustness.

Experimental results

Research questions

• RQ1 Can a self-ensemble of predictions across training epochs reliably identify correctly labeled samples under heavy label noise?
• RQ2 Does progressively filtering noisy labels during training improve generalization across standard image classification benchmarks?
• RQ3 How does SELF perform across symmetric and asymmetric label noise and across different network architectures?
• RQ4 Can the filtered-clean subset plus unsupervised loss compete with or surpass existing noise-robust methods?
• RQ5 Is the approach transferable to tasks beyond the specific networks and datasets studied?

Key findings

• SELF consistently improves robustness to both symmetric and asymmetric label noise across CIFAR-10, CIFAR-100, and ImageNet experiments.
• Using a Mean Teacher model plus a moving-average of predictions to filter labels yields more reliable identification of clean samples than single-epoch predictions.
• Progressive filtering coupled with semi-supervised learning yields higher accuracy than prior noise-aware methods across multiple architectures and noise levels.
• The approach remains effective with different network backbones, indicating architectural robustness.
• A full SELF implementation (model ensemble plus prediction ensemble) provides the strongest gains and is robust to high noise ratios (as reported in the study).
• In ablation studies, removing prediction ensemble or progressive filtering degrades performance, highlighting the contribution of each component.