[Paper Review] Identifying Mislabeled Data using the Area Under the Margin Ranking

Geoff Pleiss, Tianyi Zhang+2 more|arXiv (Cornell University)|Jan 28, 2020

Machine Learning and Data Classification71 references45 citations

TL;DR

The paper introduces Area Under the Margin (AUM) to identify mislabeled data by tracking per-sample training dynamics, and uses threshold samples with an extra class to separate mislabeled from correctly labeled data, improving test error when removing identified mislabeled samples. It demonstrates gains on WebVision, CIFAR/Tiny ImageNet, and real-world weakly-labeled datasets.

ABSTRACT

Not all data in a typical training set help with generalization; some samples can be overly ambiguous or outrightly mislabeled. This paper introduces a new method to identify such samples and mitigate their impact when training neural networks. At the heart of our algorithm is the Area Under the Margin (AUM) statistic, which exploits differences in the training dynamics of clean and mislabeled samples. A simple procedure - adding an extra class populated with purposefully mislabeled threshold samples - learns a AUM upper bound that isolates mislabeled data. This approach consistently improves upon prior work on synthetic and real-world datasets. On the WebVision50 classification task our method removes 17% of training data, yielding a 1.6% (absolute) improvement in test error. On CIFAR100 removing 13% of the data leads to a 1.2% drop in error.

Motivation & Objective

• Motivate the impact of mislabeled data on generalization in deep networks.
• Propose a simple, plug-and-play method to identify mislabeled samples from training dynamics.
• Introduce AUM (Area Under the Margin) as a per-sample metric.
• Use threshold samples via an extra class to robustly separate mislabeled data without trusted validation data.
• Demonstrate improvements by cleaning datasets across synthetic and real-world benchmarks.

Proposed method

• Define the per-sample logit margin M^(t)(x,y) between the assigned logit and the highest other logit.
• Compute AUM(x,y) as the average of margins across training epochs: AUM = (1/T) sum_t M^(t)(x,y).
• Introduce an extra artificial class (c+1) and assign a subset of threshold samples to this class to create a reference AUM distribution for mislabeled data.
• Train two networks up to the first learning-rate drop: once with the original data plus threshold samples, once with the threshold samples and the original data removed progressively based on AUM thresholds.
• Use the 99th percentile AUM of threshold samples as the threshold α to identify mislabeled data: keep data with AUM > α, remove those with AUM ≤ α.
• Repeat with different threshold samples to identify remaining mislabeled data.
• Profile identification via precision/recall on synthetic noisy data and validate by training classifiers after removing flagged samples.

Experimental results

Research questions

• RQ1 Can AUM differentiate mislabeled samples from correctly labeled ones based on training dynamics?
• RQ2 Does adding threshold samples with an extra class yield a robust, dataset-dependent threshold to separate mislabeled data without trusted validation data?
• RQ3 Do classifiers trained after removing AUM-identified samples exhibit improved test accuracy across synthetic and real-world noisy datasets?
• RQ4 Is the method robust across architectures, data augmentations, and varying noise regimes?

Key findings

• AUM captures distinct training dynamics between mislabeled and correctly labeled samples, enabling reliable ranking of data quality.
• Threshold-sample construction with an extra class provides a practical, validation-free way to set an AUM threshold.
• Across synthetic and real-world datasets, removing AUM-identified mislabeled samples improves test error, sometimes approaching or matching oracle cleaning performance.
• The method yields high precision and recall in mislabeled-data identification, outperforming several prior approaches on challenging datasets.
• On real-world weakly labeled datasets (WebVision50, Clothing100K, ImageNet-scale), AUM-based cleaning reduces error while removing a controllable fraction of data.