[Paper Review] SIGUA: Forgetting May Make Learning with Noisy Labels More Robust

Bo Han, Gang Niu+5 more|arXiv (Cornell University)|Sep 28, 2018

Machine Learning and Data Classification64 citations

TL;DR

SIGUA integrates gradient descent on likely clean data with gradient ascent on noisy data within mini-batches, enabling forgetting of undesired memorization and improving robustness to noisy labels across base learning methods.

ABSTRACT

Given data with noisy labels, over-parameterized deep networks can gradually memorize the data, and fit everything in the end. Although equipped with corrections for noisy labels, many learning methods in this area still suffer overfitting due to undesired memorization. In this paper, to relieve this issue, we propose stochastic integrated gradient underweighted ascent (SIGUA): in a mini-batch, we adopt gradient descent on good data as usual, and learning-rate-reduced gradient ascent on bad data; the proposal is a versatile approach where data goodness or badness is w.r.t. desired or undesired memorization given a base learning method. Technically, SIGUA pulls optimization back for generalization when their goals conflict with each other; philosophically, SIGUA shows forgetting undesired memorization can reinforce desired memorization. Experiments demonstrate that SIGUA successfully robustifies two typical base learning methods, so that their performance is often significantly improved.

Motivation & Objective

• Motivate the challenge of learning with noisy labels in over-parameterized deep nets.
• Introduce SIGUA as a stochastic integrated gradient underweighted ascent to balance good vs. bad data during training.
• Show how SIGUA can be integrated with existing base learning methods to improve generalization under label noise.
• Provide theoretical and empirical insights into why forgetting undesired memorization can reinforce desired memorization.

Proposed method

• Define a base learning algorithm B with its forward and backward passes.
• In each mini-batch, compute losses; classify data as good or bad using C_good and C_bad in relation to the base method.
• Accumulate positive losses for good data and underweight negative losses for bad data by a factor gamma to implement gradient ascent on bad data.
• Average the modified loss over the batch and perform a backward pass followed by an optimizer step.
• Offer computationally efficient forms, including a vectorized version where losses are weighted by masks from C_good and C_bad.
• Show how SIGUA encompasses StopGrad (gamma=0) and full ascent (gamma=1) as special cases.
• Apply SIGUA to self-teaching (sample selection) and backward correction (loss correction) to demonstrate versatility.

Experimental results

Research questions

• RQ1 How can we regularize training with noisy labels without sacrificing the ability to memorize clean data?
• RQ2 Can selective forgetting (via gradient ascent on certain data) improve generalization under label noise?
• RQ3 How does SIGUA interact with and improve existing noisy-label learning strategies like self-teaching and backward correction?
• RQ4 What are practical instantiations of good/bad data conditions for common noisy-label scenarios?
• RQ5 Does the inclusion of an underweight parameter gamma stabilize optimization when performing mixed descent/ascent updates?

Key findings

• SIGUA can significantly improve robustness over standard training and baseline noisy-label methods on MNIST and CIFAR-10 in noisy-label settings.
• In MNIST with 80% symmetric label noise, SIGUA can cause near-complete forgetting of flipped labels, boosting test accuracy from 23% to 95%.
• On CIFAR-10 with 40% open-set noise, SIGUA–BC achieves higher average test accuracy than standard BC and nnBC, with SIGUA–SL also outperforming Self-Teaching under various noise conditions.
• SIGUA encapsulates StopGrad and full gradient ascent as special cases, and the underweight parameter gamma stabilizes optimization during ascent on bad data.
• The framework is demonstrated to robustify both self-teaching (sample selection) and backward correction (loss correction), indicating broad applicability across noisy-label learning strategies.