[Paper Review] Is your noise correction noisy? PLS: Robustness to label noise with two stage detection

Paul S. Albert, Eric Arazo+3 more|arXiv (Cornell University)|Oct 10, 2022

Machine Learning and Data Classification4 citations

TL;DR

This paper proposes Pseudo-Loss Selection (PLS), a two-stage method to improve robustness to label noise by filtering incorrect pseudo-labels using a novel metric—pseudo-loss—that correlates strongly with pseudo-label correctness. By dynamically down-weighting low-confidence corrections and using a confidence-guided contrastive objective, PLS achieves state-of-the-art performance on synthetic and real-world noisy datasets, outperforming existing methods in both in-distribution and out-of-distribution noise settings.

ABSTRACT

Designing robust algorithms capable of training accurate neural networks on uncurated datasets from the web has been the subject of much research as it reduces the need for time consuming human labor. The focus of many previous research contributions has been on the detection of different types of label noise; however, this paper proposes to improve the correction accuracy of noisy samples once they have been detected. In many state-of-the-art contributions, a two phase approach is adopted where the noisy samples are detected before guessing a corrected pseudo-label in a semi-supervised fashion. The guessed pseudo-labels are then used in the supervised objective without ensuring that the label guess is likely to be correct. This can lead to confirmation bias, which reduces the noise robustness. Here we propose the pseudo-loss, a simple metric that we find to be strongly correlated with pseudo-label correctness on noisy samples. Using the pseudo-loss, we dynamically down weight under-confident pseudo-labels throughout training to avoid confirmation bias and improve the network accuracy. We additionally propose to use a confidence guided contrastive objective that learns robust representation on an interpolated objective between class bound (supervised) for confidently corrected samples and unsupervised representation for under-confident label corrections. Experiments demonstrate the state-of-the-art performance of our Pseudo-Loss Selection (PLS) algorithm on a variety of benchmark datasets including curated data synthetically corrupted with in-distribution and out-of-distribution noise, and two real world web noise datasets. Our experiments are fully reproducible github.com/PaulAlbert31/SNCF

Motivation & Objective

• To address the critical gap in label noise correction where incorrect pseudo-labels degrade model performance due to confirmation bias.
• To improve the accuracy of pseudo-label correction in two-stage label noise mitigation pipelines.
• To develop a reliable metric for identifying correct pseudo-labels, surpassing confidence-based thresholds used in prior semi-supervised methods.
• To design a contrastive objective that adapts between supervised and unsupervised learning based on pseudo-label reliability.
• To validate the method on both synthetically corrupted datasets and real-world web-crawled datasets with diverse noise types.

Proposed method

• Proposes the pseudo-loss—a new metric that quantifies the likelihood of a pseudo-label being correct, showing strong correlation with actual correctness.
• Uses the pseudo-loss to dynamically down-weight training updates for low-confidence pseudo-labels, mitigating confirmation bias during optimization.
• Introduces a confidence-guided interpolated contrastive loss that combines supervised contrastive learning for reliable pseudo-labels and unsupervised contrastive learning for unreliable ones.
• Applies a two-stage framework: first detecting noisy samples using existing methods, then filtering incorrect corrections using the pseudo-loss.
• Employs a mixture model on training losses to detect noisy samples in the first stage, followed by pseudo-label generation via consistency regularization.
• Trains a neural network using a joint objective combining cross-entropy classification and the interpolated contrastive loss to learn robust representations.

Experimental results

Research questions

• RQ1 Can a metric other than prediction confidence reliably identify correct pseudo-labels in noisy label correction?
• RQ2 Does filtering out incorrect pseudo-labels using the pseudo-loss reduce confirmation bias and improve model accuracy on noisy datasets?
• RQ3 How does the proposed interpolated contrastive objective improve representation learning when some pseudo-labels are incorrect?
• RQ4 Does the PLS method generalize to both in-distribution and out-of-distribution noise in real-world web-crawled datasets?
• RQ5 Can PLS achieve state-of-the-art performance without relying on ensemble methods or label smoothing?

Key findings

• PLS achieves state-of-the-art accuracy on CIFAR-100 with synthetic in-distribution noise, outperforming existing methods even without label smoothing or ensemble techniques.
• On CIFAR-100 with mixed in-distribution and out-of-distribution noise, PLS effectively filters out OOD images by assigning them low pseudo-loss values, preventing incorrect corrections.
• In controlled web corruption experiments (CNWL), PLS achieves superior performance compared to SOTA methods like SNCF, even with a single network and no self-supervised pre-training.
• On real-world web-crawled datasets (Web-aircraft, Web-bird, Web-car), PLS matches or exceeds the performance of ensemble and co-learning methods, demonstrating strong generalization.
• The pseudo-loss metric significantly outperforms traditional confidence-based thresholds in identifying correct pseudo-labels, as validated across multiple benchmarks and noise types.
• Ablation studies confirm that the interpolated contrastive objective and pseudo-loss filtering are both essential components for PLS’s performance gains.