[Paper Review] An Ensemble Noise-Robust K-fold Cross-Validation Selection Method for Noisy Labels

Yong Wen, Marcus Kalander+2 more|arXiv (Cornell University)|Jul 6, 2021

Machine Learning and Data Classification27 references4 citations

TL;DR

This paper proposes E-NKCVS, an ensemble noise-robust K-fold cross-validation method that selects clean samples from noisy datasets by leveraging model consensus and entropy-based label re-weighting. It outperforms state-of-the-art methods on image and text classification tasks, achieving 75.0% accuracy on Clothing-1M and 77.6% on WebVision, with consistent gains under high noise ratios and minimal degradation on clean data.

ABSTRACT

We consider the problem of training robust and accurate deep neural networks (DNNs) when subject to various proportions of noisy labels. Large-scale datasets tend to contain mislabeled samples that can be memorized by DNNs, impeding the performance. With appropriate handling, this degradation can be alleviated. There are two problems to consider: how to distinguish clean samples and how to deal with noisy samples. In this paper, we present Ensemble Noise-robust K-fold Cross-Validation Selection (E-NKCVS) to effectively select clean samples from noisy data, solving the first problem. For the second problem, we create a new pseudo label for any sample determined to have an uncertain or likely corrupt label. E-NKCVS obtains multiple predicted labels for each sample and the entropy of these labels is used to tune the weight given to the pseudo label and the given label. Theoretical analysis and extensive verification of the algorithms in the noisy label setting are provided. We evaluate our approach on various image and text classification tasks where the labels have been manually corrupted with different noise ratios. Additionally, two large real-world noisy datasets are also used, Clothing-1M and WebVision. E-NKCVS is empirically shown to be highly tolerant to considerable proportions of label noise and has a consistent improvement over state-of-the-art methods. Especially on more difficult datasets with higher noise ratios, we can achieve a significant improvement over the second-best model. Moreover, our proposed approach can easily be integrated into existing DNN methods to improve their robustness against label noise.

Motivation & Objective

• To address the challenge of training deep neural networks on datasets with high proportions of noisy labels, which degrade model generalization.
• To develop a method that effectively identifies and selects clean samples from noisy data without requiring auxiliary clean data.
• To design a label re-weighting scheme that dynamically adjusts confidence between ground-truth and pseudo-labels based on prediction uncertainty.
• To ensure the method is robust to varying noise levels and easily integrable into existing DNN frameworks.

Proposed method

• E-NKCVS uses K-fold cross-validation to train multiple models on different data splits, aggregating predictions across folds to assess label consistency.
• Samples with consistent predicted labels across folds are selected as likely clean; those with inconsistent predictions are flagged as potentially noisy.
• For noisy or uncertain samples, a hybrid label is formed by combining the given (noisy) label and a pseudo-label generated from ensemble predictions.
• The weight assigned to the pseudo-label is determined by the entropy of the ensemble predictions—lower entropy implies higher confidence in the pseudo-label.
• Mixup data augmentation is applied during training to improve robustness and generalization.
• The method is designed to be plug-and-play, compatible with existing DNN architectures and loss functions.

Experimental results

Research questions

• RQ1 Can an ensemble K-fold cross-validation approach effectively identify clean samples in datasets with high label noise?
• RQ2 How does entropy-based label re-weighting improve model performance when ground-truth labels are unreliable?
• RQ3 Does E-NKCVS maintain high performance on clean data while significantly improving accuracy under high noise ratios?
• RQ4 How does E-NKCVS compare to state-of-the-art methods on real-world noisy datasets like Clothing-1M and WebVision?

Key findings

• E-NKCVS achieves 75.0% test accuracy on Clothing-1M, surpassing the prior state-of-the-art and outperforming standard cross-entropy loss (69.0%).
• On (mini) WebVision, E-NKCVS attains 77.6% accuracy, slightly improving upon the prior state-of-the-art and outperforming strong baselines like DivideMix (77.3%).
• On CIFAR-100 with 40% asymmetric label noise, E-NKCVS achieves a significant improvement over the second-best model, demonstrating strong robustness on challenging datasets.
• When no label noise is present (ε=0), E-NKCVS maintains test accuracy nearly identical to standard cross-entropy, indicating minimal negative impact on clean data.
• The method shows consistent performance gains across multiple datasets and noise types, including synthetic symmetric and asymmetric noise, and real-world label noise.
• An extensive hyperparameter analysis confirms the method’s robustness and versatility across different network architectures and training settings.