[Paper Review] Reliable Label Correction is a Good Booster When Learning with Extremely Noisy Labels

Kai Wang, Xiangyu Peng+5 more|arXiv (Cornell University)|Apr 30, 2022

Machine Learning and Data Classification4 citations

TL;DR

This paper proposes LC-Booster, a novel framework that combines reliable label correction with sample selection to improve deep learning under extreme label noise. By leveraging a purified clean set from sample selection to enable accurate label correction, LC-Booster expands the clean set size while maintaining high label precision, achieving 92.9% accuracy on CIFAR-10 and 48.4% on CIFAR-100 under 90% symmetric noise—surpassing state-of-the-art methods.

ABSTRACT

Learning with noisy labels has aroused much research interest since data annotations, especially for large-scale datasets, may be inevitably imperfect. Recent approaches resort to a semi-supervised learning problem by dividing training samples into clean and noisy sets. This paradigm, however, is prone to significant degeneration under heavy label noise, as the number of clean samples is too small for conventional methods to behave well. In this paper, we introduce a novel framework, termed as LC-Booster, to explicitly tackle learning under extreme noise. The core idea of LC-Booster is to incorporate label correction into the sample selection, so that more purified samples, through the reliable label correction, can be utilized for training, thereby alleviating the confirmation bias. Experiments show that LC-Booster advances state-of-the-art results on several noisy-label benchmarks, including CIFAR-10, CIFAR-100, Clothing1M and WebVision. Remarkably, under the extreme 90\% noise ratio, LC-Booster achieves 92.9\% and 48.4\% accuracy on CIFAR-10 and CIFAR-100, surpassing state-of-the-art methods by a large margin.

Motivation & Objective

• To address the performance degradation of existing sample selection methods under extreme label noise, where the number of clean samples becomes too small for effective training.
• To investigate whether label correction can be reliably applied in the sample selection setting to expand the size of the clean set without introducing confirmation bias.
• To develop a framework that jointly leverages sample selection and label correction to improve model generalization under extreme noise.
• To demonstrate that reliable label correction, when applied on a purified clean set, can significantly boost performance without requiring external clean anchors or bootstrapping.

Proposed method

• The framework begins with a warm-up phase to learn basic patterns before applying Gaussian Mixture Model (GMM) loss-based sample selection to separate clean and noisy samples.
• Clean samples are trained with supervised loss, while noisy samples are treated in an unsupervised manner using a hybrid augmentation (H-Aug.) strategy to enforce consistency across weak-weak and weak-strong views.
• At the mid-training stage, Reliable Label Correction (ReCo) is applied to revise labels of both clean and noisy samples using predictions from a model trained on the purified clean set.
• The ReCo mechanism uses a thresholded confidence score to ensure high-precision label revisions, with theoretical justification for label precision under proper thresholding.
• The method avoids confirmation bias by relying on a highly purified clean set, which provides reliable model predictions for label correction.
• The framework dynamically expands the clean set size after ReCo, significantly increasing the number of high-quality training samples.

Experimental results

Research questions

• RQ1 Can label correction be reliably applied in a sample selection framework to expand the clean set size without introducing error accumulation?
• RQ2 Does combining sample selection with label correction lead to improved generalization under extreme label noise (e.g., 90% noise)?
• RQ3 How does the performance of LC-Booster compare to state-of-the-art methods under increasing noise levels, especially beyond 90%?
• RQ4 Can the precision of label correction be theoretically guaranteed when applied on a purified clean set?
• RQ5 Does performing label correction only once during training suffice for optimal performance?

Key findings

• LC-Booster achieves 92.9% test accuracy on CIFAR-10 and 48.4% on CIFAR-100 under 90% symmetric label noise, significantly outperforming prior state-of-the-art methods.
• The re-labeling precision of LC-Booster exceeds 90% under 90% symmetric noise, demonstrating its superior ability to cleanse noisy labels.
• The size of the clean set in LC-Booster increases substantially after label correction, surpassing DivideMix by a large margin and contributing to improved performance.
• LC-Booster maintains higher AUC in clean sample selection throughout training, indicating more precise identification of clean samples compared to baseline methods.
• The performance gap between LC-Booster and DivideMix grows as noise increases from 90% to 93%, confirming its robustness under extreme noise.
• Re-labeling once at the 100th epoch is sufficient; further re-labeling does not improve accuracy, indicating optimal timing for label correction.