[Paper Review] Learning with Feature-Dependent Label Noise: A Progressive Approach

Yikai Zhang, Songzhu Zheng+3 more|arXiv (Cornell University)|Mar 13, 2021

Machine Learning and Data Classification37 references32 citations

TL;DR

Introduces Progressive Label Correction (PLC) to handle Poly-Margin Diminishing (PMD) label noise with theoretical guarantees to converge to Bayes-optimal classifier, plus strong empirical results.

ABSTRACT

Label noise is frequently observed in real-world large-scale datasets. The noise is introduced due to a variety of reasons; it is heterogeneous and feature-dependent. Most existing approaches to handling noisy labels fall into two categories: they either assume an ideal feature-independent noise, or remain heuristic without theoretical guarantees. In this paper, we propose to target a new family of feature-dependent label noise, which is much more general than commonly used i.i.d. label noise and encompasses a broad spectrum of noise patterns. Focusing on this general noise family, we propose a progressive label correction algorithm that iteratively corrects labels and refines the model. We provide theoretical guarantees showing that for a wide variety of (unknown) noise patterns, a classifier trained with this strategy converges to be consistent with the Bayes classifier. In experiments, our method outperforms SOTA baselines and is robust to various noise types and levels.

Motivation & Objective

• Motivate learning under heterogeneous, feature-dependent label noise beyond i.i.d. assumptions.
• Introduce the PMD noise family to generalize real-world label corruption patterns.
• Develop a data-recalibrating progressive-label-correction algorithm with convergence guarantees.
• Provide theoretical analysis showing convergence to the Bayes classifier under PMD noise.
• Demonstrate empirical superiority over SOTA baselines on standard benchmarks and real data.

Proposed method

• Define PMD (Poly-Margin Diminishing) label noise with margin t0 and bounds that apply away from the decision boundary.
• Propose Progressive Label Correction (PLC): warm-up training, then iteratively correct labels using a high-confidence classifier, lowering the threshold over time.
• Alternate label correction and model retraining to progressively improve label purity and model accuracy.
• Extend PLC to multi-class by using the top prediction gap for correction decisions.
• Present algorithmic steps in a formal algorithm (Algorithm 1) with parameter choices and stopping criteria.
• Provide theoretical guarantees (Theorem 1 and supporting lemmas) that PLC improves label purity and converges to η*.

Experimental results

Research questions

• RQ1 Can a data-recalibrating method with PMD noise provide provable convergence to the Bayes optimal classifier?
• RQ2 How does PMD noise generalize common noise models, and can PLC robustly correct labels under such noise?
• RQ3 What are the theoretical conditions under which PLC ensures improvement per iteration and overall convergence?
• RQ4 How does PLC perform empirically against state-of-the-art noisy-label methods on CIFAR-10/100 with various noise patterns and on real-world data like Clothing1M?
• RQ5 Is PLC robust to hyperparameter choices for thresholding and correction pace?

Key findings

• PLC achieves state-of-the-art accuracy on CIFAR-10/100 across several PMD noise types at 35% and 70% corruption levels.
• Under hybrid noise (feature-dependent + i.i.d.), PLC outperforms baselines, maintaining robustness across noisy settings.
• On Clothing1M, PLC demonstrates competitive accuracy against multiple baselines using a realistic large-scale noisy dataset.
• Theoretical results (Theorem 1) show that, with PMD noise, PLC yields high-probability convergence to the Bayes optimal classifier, given mild assumptions.
• A warm-up phase plus progressively tightened correction threshold guarantees expansion of the “pure” region and overall label-purity improvement.