[Paper Review] Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels

Pengfei Chen, Benben Liao+2 more|arXiv (Cornell University)|May 13, 2019

Machine Learning and Data ClassificationComputer Science27 references139 citations

TL;DR

The paper analyzes how label noise affects DNN generalization, proves test accuracy is a quadratic function of symmetric noise, and proposes INCV+Co-teaching to identify clean samples and train robust models, achieving state-of-the-art results on CIFAR-10 and WebVision.

ABSTRACT

Noisy labels are ubiquitous in real-world datasets, which poses a challenge for robustly training deep neural networks (DNNs) as DNNs usually have the high capacity to memorize the noisy labels. In this paper, we find that the test accuracy can be quantitatively characterized in terms of the noise ratio in datasets. In particular, the test accuracy is a quadratic function of the noise ratio in the case of symmetric noise, which explains the experimental findings previously published. Based on our analysis, we apply cross-validation to randomly split noisy datasets, which identifies most samples that have correct labels. Then we adopt the Co-teaching strategy which takes full advantage of the identified samples to train DNNs robustly against noisy labels. Compared with extensive state-of-the-art methods, our strategy consistently improves the generalization performance of DNNs under both synthetic and real-world training noise.

Motivation & Objective

• Quantify how label noise impacts the generalization of deep neural networks trained with noisy labels.
• Provide a theoretical framework relating test accuracy to noise ratio under symmetric and asymmetric noise.
• Develop practical algorithms to identify clean labels in noisy datasets and integrate them into robust training.
• Demonstrate state-of-the-art performance on synthetic and real-world noisy datasets (CIFAR-10 and WebVision) compared to existing methods.

Proposed method

• Model training with noisy labels using a noise transition matrix T to analyze generalization in distribution.
• Derive the test-accuracy formulas for symmetric and asymmetric noise (e.g., Eqs. 4–5 and related corollaries).
• Introduce Noisy Cross-Validation (NCV) to identify clean samples by cross-training on split halves (Algorithm 1).
• Define Iterative Noisy Cross-Validation (INCV) to iteratively enlarge a clean subset and estimate noise (Algorithm 2).
• Integrate INCV with a strengthened Co-teaching framework (Algorithm 3) to train two networks with clean-subset emphasis and controlled use of candidate samples.

Experimental results

Research questions

• RQ1 How does label noise (symmetric vs. asymmetric) quantitatively affect DNN test accuracy and generalization?
• RQ2 Can the test accuracy of DNNs trained with noisy labels be expressed as a function of the noise ratio, and does it match empirical observations?
• RQ3 Can we reliably identify clean (correct) samples from a noisy dataset to improve learning?
• RQ4 Does combining Iterative Noisy Cross-Validation with Co-teaching yield superior robustness and accuracy on both synthetic and real-world noisy datasets?

Key findings

• Test accuracy is a quadratic function of the noise ratio under symmetric noise.
• Predictions of DNNs trained on noisy data follow the same distribution as the true labels characterized by the noise transition matrix T (generalization in distribution).
• Noisy Cross-Validation (NCV) and Iterative NCV (INCV) can identify a high fraction of clean samples and reduce the effective noise in the selected set (LP and LR metrics).
• The INCV-enhanced Co-teaching framework achieves superior test accuracy on CIFAR-10 with symmetric and asymmetric noise and on WebVision/ILSVRC12 validation compared to multiple baselines.
• Empirical results show M ≈ T in confusion matrices for networks trained on noisy CIFAR-10, validating the theoretical predictions.