[Paper Review] SELC: Self-Ensemble Label Correction Improves Learning with Noisy Labels

Yangdi Lu, Wenbo He|arXiv (Cornell University)|May 2, 2022

Machine Learning and Data Classification4 citations

TL;DR

SELC is a simple yet effective method for training deep neural networks with noisy labels by using self-ensemble predictions from early training epochs to progressively correct noisy labels. By leveraging exponential moving averages of model outputs before the memorization turning point, SELC reduces overfitting to mislabeled data and achieves state-of-the-art performance across symmetric, class-conditional, and instance-dependent noise settings, with minimal hyperparameter sensitivity.

ABSTRACT

Deep neural networks are prone to overfitting noisy labels, resulting in poor generalization performance. To overcome this problem, we present a simple and effective method self-ensemble label correction (SELC) to progressively correct noisy labels and refine the model. We look deeper into the memorization behavior in training with noisy labels and observe that the network outputs are reliable in the early stage. To retain this reliable knowledge, SELC uses ensemble predictions formed by an exponential moving average of network outputs to update the original noisy labels. We show that training with SELC refines the model by gradually reducing supervision from noisy labels and increasing supervision from ensemble predictions. Despite its simplicity, compared with many state-of-the-art methods, SELC obtains more promising and stable results in the presence of class-conditional, instance-dependent, and real-world label noise. The code is available at https://github.com/MacLLL/SELC.

Motivation & Objective

• To address the problem of deep neural networks overfitting to noisy labels during training.
• To identify and exploit the early-stage memorization behavior where model predictions on clean samples are reliable.
• To develop a label correction method that uses historical model predictions to refine supervision signals without requiring test data or noise rate information.
• To create a robust, simple, and effective approach that can be easily integrated with existing training techniques like mixup.

Proposed method

• SELC uses an exponential moving average (EMA) of model predictions from early training epochs to form a self-ensemble prediction, serving as a more reliable target than noisy labels.
• The method dynamically corrects the original noisy labels by combining them with the EMA-based ensemble predictions using a momentum-weighted update.
• A novel unsupervised metric based on loss modeling is proposed to estimate the turning point in training—when memorization of mislabeled samples begins—without requiring test data or noise rate information.
• The correction process gradually reduces reliance on noisy labels and increases reliance on the more stable ensemble predictions, improving generalization.
• The approach is compatible with data augmentation techniques like mixup, which further enhances performance when integrated.

Experimental results

Research questions

• RQ1 Can early, consistent model predictions from clean samples be leveraged to correct noisy labels during training?
• RQ2 How can the turning point in training—when memorization of mislabeled samples begins—be detected without access to test data or noise rate?
• RQ3 Can a self-ensemble of past model predictions improve label correction and model generalization under various noise types?
• RQ4 How does SELC compare to state-of-the-art methods in terms of robustness and stability across different noise levels and settings?

Key findings

• SELC achieves state-of-the-art performance on CIFAR-10 with symmetric, class-conditional, and instance-dependent label noise, outperforming existing methods across all noise ratios.
• On CIFAR-10 with 80% symmetric noise, SELC improves test accuracy by over 10% compared to standard cross-entropy training.
• With 70% PMD noise on CIFAR-100, SELC achieves a ~10% accuracy improvement over baselines, demonstrating strong robustness under high noise levels.
• On real-world datasets, SELC outperforms baselines on ANIMAL-10N and Webvision, and achieves comparable performance to SOTA methods like PLC on Clothing1M despite its simplicity.
• The correction accuracy of SELC remains stable and high across training epochs, indicating reliable and consistent label correction.
• SELC shows mild sensitivity to the hyperparameter α, with minimal performance drop across α values of 0.85, 0.9, and 0.95, indicating robustness to hyperparameter choice.