[Paper Review] Combating Label Noise in Deep Learning Using Abstention

Sunil Thulasidasan, Tanmoy Bhattacharya+3 more|arXiv (Cornell University)|May 27, 2019

Machine Learning and Data ClassificationComputer Science4 references77 citations

TL;DR

The paper introduces a deep abstaining classifier (DAC) that abstains on confusing samples during training and inference to robustly handle label noise, including structured and unstructured noise, with auto-tuned abstention and data-cleaning capabilities.

ABSTRACT

We introduce a novel method to combat label noise when training deep neural networks for classification. We propose a loss function that permits abstention during training thereby allowing the DNN to abstain on confusing samples while continuing to learn and improve classification performance on the non-abstained samples. We show how such a deep abstaining classifier (DAC) can be used for robust learning in the presence of different types of label noise. In the case of structured or systematic label noise -- where noisy training labels or confusing examples are correlated with underlying features of the data-- training with abstention enables representation learning for features that are associated with unreliable labels. In the case of unstructured (arbitrary) label noise, abstention during training enables the DAC to be used as an effective data cleaner by identifying samples that are likely to have label noise. We provide analytical results on the loss function behavior that enable dynamic adaption of abstention rates based on learning progress during training. We demonstrate the utility of the deep abstaining classifier for various image classification tasks under different types of label noise; in the case of arbitrary label noise, we show significant improvements over previously published results on multiple image benchmarks. Source code is available at https://github.com/thulas/dac-label-noise

Motivation & Objective

• Motivate and address the impact of label noise on deep learning classification performance.
• Propose a loss function that enables abstention during training without sacrificing learning on non-abstained samples.
• Show that abstention aids robust learning under structured noise and can act as an effective data cleaner under unstructured noise.
• Demonstrate the method’s applicability across image benchmarks and provide insights into abstention behavior and representations.

Proposed method

• Introduce a k+1 output DAC where the extra output models abstention.
• Define a modified cross-entropy loss that combines standard class probabilities with an abstention term and a tunable abstention penalty α.
• Prove that learning on true classes is preserved under abstention (gradient behavior).
• Provide an auto-tuning scheme for α during training to balance abstention and learning progress.
• Demonstrate structured-noise learning by enabling the DAC to learn feature mappings associated with unreliable labels and abstain accordingly.
• Demonstrate unstructured-noise data cleaning by using DAC to identify and prune likely noisy samples before retraining a clean DNN.

Experimental results

Research questions

• RQ1 Can abstention during training help a DNN learn robust representations when labels are systematically noisy?
• RQ2 Can the DAC identify samples with unreliable labels and abstain on them during training?
• RQ3 Can abstention training serve as an effective data-cleaning step for downstream learning under uniform label noise?
• RQ4 How does the abstention mechanism influence learning dynamics and memorization in DNNs?

Key findings

• DAC enables robust learning in presence of label noise across structured and unstructured scenarios.
• The DAC can learn features indicative of unreliable labels and abstain with high precision and recall on such samples.
• Using DAC to filter noisy samples and retrain a downstream model yields notable accuracy gains over baselines and other noise-robust methods.
• DAC abstention behavior evolves with training, reducing abstention as learning progresses, and can reveal interpretable feature associations (e.g., artifacts like smudges).
• In data-cleaning experiments, pruning noisy samples via DAC followed by training on the cleaner set often surpasses state-of-the-art noise-handling methods.