[Paper Review] Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels

Curtis G. Northcutt, Tailin Wu+1 more|arXiv (Cornell University)|May 4, 2017

Machine Learning and Data Classification68 citations

TL;DR

Rank Pruning (RP) offers a time-efficient solution for binary classification with noisy labels by estimating asymmetric noise rates and pruning mislabeled examples using confident predictions, achieving equivalent risk to clean-label learning under ideal conditions.

ABSTRACT

Noisy PN learning is the problem of binary classification when training examples may be mislabeled (flipped) uniformly with noise rate rho1 for positive examples and rho0 for negative examples. We propose Rank Pruning (RP) to solve noisy PN learning and the open problem of estimating the noise rates, i.e. the fraction of wrong positive and negative labels. Unlike prior solutions, RP is time-efficient and general, requiring O(T) for any unrestricted choice of probabilistic classifier with T fitting time. We prove RP has consistent noise estimation and equivalent expected risk as learning with uncorrupted labels in ideal conditions, and derive closed-form solutions when conditions are non-ideal. RP achieves state-of-the-art noise estimation and F1, error, and AUC-PR for both MNIST and CIFAR datasets, regardless of the amount of noise and performs similarly impressively when a large portion of training examples are noise drawn from a third distribution. To highlight, RP with a CNN classifier can predict if an MNIST digit is a "one"or "not" with only 0.25% error, and 0.46 error across all digits, even when 50% of positive examples are mislabeled and 50% of observed positive labels are mislabeled negative examples.

Motivation & Objective

• Motivate and formalize the problem of binary classification with asymmetric label noise ($ ilde{P} ilde{N}$ learning).
• Propose Rank Pruning (RP) to estimate noise rates $ ho_1$ and $ ho_0$ and prune mislabeled examples.
• Show that RP achieves consistent noise estimation and equivalent expected risk to clean-label learning under ideal conditions, with closed-form solutions for non-ideal cases.
• Demonstrate RP’s scalability and robustness across datasets (MNIST, CIFAR) and classifiers (LR, CNN).
• Provide theoretical guarantees and empirical evidence that RP improves F1, AUC-PR, and error without sacrificing efficiency.

Proposed method

• Estimate asymmetric noise rates $ ho_1$ and $ ho_0$ from confident counts of predicted labels, yielding a robust, out-of-sample estimation. 1
• Compute LB and UB thresholds from model predictions to partition corrupted data into confidently correct/incorrect subsets.
• Prune the corresponding fractions of $ ilde{P}$ and $ ilde{N}$ using BFPRT to obtain $ ilde{P}_{conf}$ and $ ilde{N}_{conf}$ in $\,O(n)$ time.
• Refit the classifier on the confident subset with class-conditioned weighting to counterbalance estimated noise.
• Prove that under range separability, RP yields equivalent expected risk to learning from clean labels (Theorem 5).
• Provide closed-form expressions for the non-ideal case and establish consistency under ideal/unideal conditions (Lemmas/Theorems).

Experimental results

Research questions

• RQ1 Can RP accurately estimate the asymmetric noise rates $ ho_1$ and $ ho_0$ from corrupted labels?
• RQ2 Does RP yield an equivalent expected risk to learning with clean labels under ideal conditions?
• RQ3 How does RP perform when $g$ is not ideal or when $P$ and $N$ have overlapping support?
• RQ4 Is RP time-efficient and scalable to large datasets and complex classifiers?
• RQ5 Does RP improve standard metrics (F1, AUC-PR, Error) across MNIST, CIFAR, and varying noise levels?

Key findings

• RP achieves robust, time-efficient noise-rate estimation with $ ilde{P} ilde{N}$ data.
• Under ideal conditions, RP’s noise estimates are exact and the resulting risk matches learning with true labels.
• RP remains robust even when $g$ is imperfect, with closed-form adjustments for non-ideal scenarios.
• Empirically, RP attains state-of-the-art F1, AUC-PR, and Error on MNIST and CIFAR across varying noise levels and added noise from a third distribution.
• With CNNs on MNIST, RP can reach 0.25% error for one-vs-not-one and 0.46% error overall under substantial label noise (e.g., 50% mislabels).
• RP scales with data size, feature dimension, and noise fractions, while maintaining O(T) fitting time and linear pruning complexity.