[Paper Review] No Regret Sample Selection with Noisy Labels

Song, H., Nariaki Mitsuo+2 more|arXiv (Cornell University)|Mar 6, 2020

Machine Learning and Data Classification39 references4 citations

TL;DR

This paper proposes TA kS, a novel deep learning method that uses adaptive k-set selection to suppress total selection risk in training with noisy labels, achieving state-of-the-art accuracy with significantly lower computational cost. By theoretically bounding regret and leveraging perturbation-based selection, TA kS dynamically selects clean samples while training a single DNN, outperforming co-training and other baselines in accuracy and efficiency across multiple noisy datasets.

ABSTRACT

Deep neural networks (DNNs) suffer from noisy-labeled data because of the risk of overfitting. To avoid the risk, in this paper, we propose a novel DNN training method with sample selection based on adaptive k-set selection, which selects k (< n) clean sample candidates from the whole n noisy training samples at each epoch. It has a strong advantage of guaranteeing the performance of the selection theoretically. Roughly speaking, a regret, which is defined by the difference between the actual selection and the best selection, of the proposed method is theoretically bounded, even though the best selection is unknown until the end of all epochs. The experimental results on multiple noisy-labeled datasets demonstrate that our sample selection strategy works effectively in the DNN training; in fact, the proposed method achieved the best or the second-best performance among state-of-the-art methods, while requiring a significantly lower computational cost. The code is available at https://github.com/songheony/TAkS.

Motivation & Objective

• To address the challenge of training deep neural networks on datasets with noisy labels, which can lead to overfitting and degraded performance.
• To develop a sample selection method that can effectively identify and exclude incorrectly labeled samples during training.
• To provide a theoretically grounded approach with bounded regret, even when the optimal selection is unknown in advance.
• To achieve high performance with lower computational cost compared to existing co-training and exhaustive k-set methods.
• To demonstrate the effectiveness of adaptive k-set selection via Follow-the-Perturbed-Leader (FPL) in online, uncertain training environments.

Proposed method

• The method uses adaptive k-set selection to choose k samples with minimal noise-risk at each training epoch, based on a noise-risk vector θt estimated from model predictions.
• It formulates the selection problem as minimizing total selection risk ∑ₜ dₜ·θₜ over T epochs, where dₜ is the k-hot selection vector at epoch t.
• The FPL algorithm introduces random perturbations to the noise-risk vector, enabling online selection with a theoretical regret bound, even when θₜ is unreliable.
• The method trains a single DNN on the selected k samples per epoch, avoiding the computational overhead of co-training with multiple networks.
• Noise-risk θi,t is computed as the likelihood that sample i is incorrectly labeled, updated iteratively during training.
• The approach is designed to be computationally efficient, with O(n) time complexity per epoch, enabling scalability to large datasets.

Experimental results

Research questions

• RQ1 Can adaptive k-set selection with regret minimization improve DNN training on noisy-labeled datasets compared to existing sample selection methods?
• RQ2 Does the FPL-based selection strategy achieve a bounded regret despite the online and uncertain nature of noise-risk estimation?
• RQ3 Can a single DNN trained on dynamically selected k samples outperform co-training methods that use coupled networks?
• RQ4 How does the method perform under varying levels of label noise, especially in extreme cases like 80% symmetric noise?
• RQ5 To what extent does the noise-risk vector successfully identify and exclude mislabeled samples during training?

Key findings

• TA kS achieved the best or second-best test accuracy across all benchmark datasets, including MNIST, CIFAR-10, CIFAR-100, and Clothing1M, under various noise settings.
• On CIFAR-10 with symmetric 80% noise, TA kS achieved 70.22% test accuracy, significantly outperforming Greedy (70.22%) and Naive (66.06%) in the ablation study.
• The method maintained or improved label precision throughout training, even under high noise, indicating strong noise tolerance and stable learning.
• Feature visualization showed that clean samples formed tight clusters, while mislabeled samples were scattered and mostly excluded by the selection mechanism.
• TA kS was faster than standard training and co-training baselines—up to four times faster on symmetric 80% noise due to using only k < n samples per epoch.
• The ablation study confirmed that FPL’s perturbation mechanism is essential, as Greedy and Naive failed under high noise due to unstable risk estimates.