[Paper Review] Sample Selection with Uncertainty of Losses for Learning with Noisy Labels

Xiaobo Xia, Tongliang Liu+5 more|arXiv (Cornell University)|Jun 1, 2021

Machine Learning and Data Classification69 references48 citations

TL;DR

The paper introduces CNLCU, a sample selection method that uses interval-based uncertainty in losses to robustly train with noisy labels, improving robustness across balanced/imbalanced and real-world noise.

ABSTRACT

In learning with noisy labels, the sample selection approach is very popular, which regards small-loss data as correctly labeled during training. However, losses are generated on-the-fly based on the model being trained with noisy labels, and thus large-loss data are likely but not certainly to be incorrect. There are actually two possibilities of a large-loss data point: (a) it is mislabeled, and then its loss decreases slower than other data, since deep neural networks "learn patterns first"; (b) it belongs to an underrepresented group of data and has not been selected yet. In this paper, we incorporate the uncertainty of losses by adopting interval estimation instead of point estimation of losses, where lower bounds of the confidence intervals of losses derived from distribution-free concentration inequalities, but not losses themselves, are used for sample selection. In this way, we also give large-loss but less selected data a try; then, we can better distinguish between the cases (a) and (b) by seeing if the losses effectively decrease with the uncertainty after the try. As a result, we can better explore underrepresented data that are correctly labeled but seem to be mislabeled at first glance. Experiments demonstrate that the proposed method is superior to baselines and robust to a broad range of label noise types.

Motivation & Objective

• Motivate robust learning under label noise where small-loss selection may be unreliable.
• Incorporate loss uncertainty using interval estimates rather than point losses.
• Develop robust mean estimators (soft and hard truncation) to aggregate losses over time.
• Encourage selecting less-sampled but potentially correctly labeled data to improve generalization.
• Demonstrate effectiveness across synthetic balanced/imbalanced and real-world noisy datasets.

Proposed method

• Model training loss as a time-evolving (Markov) process across iterations.
• Extend time intervals and aggregate losses over multiple iterations to stabilize selection.
• Introduce soft truncation using a robust mean estimator with a log-based influence function.
• Introduce hard truncation with KNN-based outlier removal for robust mean estimation.
• Derive concentration bounds for soft and hard estimators to obtain conservative selection criteria.
• Use a two-network co-training framework where each network selects a subset of examples for its Peer to train on (Algorithm 1 CNLCU).

Experimental results

Research questions

• RQ1 Can loss uncertainty be leveraged to improve sample selection under noisy labels?
• RQ2 Do robust mean estimators and conservative bounds enhance robustness to various noise types and class imbalance?
• RQ3 How does CNLCU compare to existing sample-selection methods on synthetic and real-world noisy datasets?
• RQ4 Is it beneficial to explore large-loss but under-selected data to recover underrepresented clean examples?
• RQ5 How do soft and hard truncation strategies perform under different training intervals and noise regimes?

Key findings

• CNLCU-S and CNLCU-H achieve superior or competitive accuracy across MNIST, F-MNIST, and CIFAR datasets under multiple noise types and levels.
• The proposed methods show robustness to imbalanced noisy data and broad noise types, outperforming several baselines in key settings.
• Soft and hard truncation improve stability of loss-based sample selection through robust mean estimation and outlier removal.
• CNLCU yields notable gains on imbalanced synthetic datasets, indicating better utilization of underrepresented classes.
• Experiments on Clothing1M show CNLCU variants outperforming JoCor on Best and Last metrics, though not always reaching the very best state-of-the-art backbones.