[Paper Review] Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels

Bo Han, Quanming Yao+6 more|arXiv (Cornell University)|Apr 18, 2018

Machine Learning and Data Classification44 references453 citations

TL;DR

Co-teaching trains two neural networks simultaneously and lets them teach each other by exchanging small-loss, potentially clean, samples per mini-batch to robustly learn under extremely noisy labels.

ABSTRACT

Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally memorize these noisy labels sooner or later during training. Nonetheless, recent studies on the memorization effects of deep neural networks show that they would first memorize training data of clean labels and then those of noisy labels. Therefore in this paper, we propose a new deep learning paradigm called Co-teaching for combating with noisy labels. Namely, we train two deep neural networks simultaneously, and let them teach each other given every mini-batch: firstly, each network feeds forward all data and selects some data of possibly clean labels; secondly, two networks communicate with each other what data in this mini-batch should be used for training; finally, each network back propagates the data selected by its peer network and updates itself. Empirical results on noisy versions of MNIST, CIFAR-10 and CIFAR-100 demonstrate that Co-teaching is much superior to the state-of-the-art methods in the robustness of trained deep models.

Motivation & Objective

• Motivate robust learning when training labels are highly corrupted in real-world data.
• Propose a simple dual-network training paradigm that mitigates memorization of noisy labels.
• Leverage memorization dynamics of deep networks to prioritize clean samples via small-loss sampling.
• Demonstrate robustness of the method across synthetic noise settings on standard benchmarks.

Proposed method

• Maintain two networks f and g with parameters w_f and w_g and train them in a cross-updating, mini-batch regime.
• In each mini-batch, each network selects a subset of small-loss instances (R(T)% of the batch) as potentially clean.
• Each network updates its parameters using the data selected by the peer network (f updated by g’s small-loss set; g updated by f’s small-loss set).
• R(T) is a non-increasing schedule that starts near 1 and decays to drop more noisy examples over time.
• The intuition relies on memorization dynamics: deep nets learn clean, easy patterns first and gradually memorize noise, so small-loss samples are likelier to be clean early on.
• Two networks enable cross-filtering of potentially noisy instances, reducing error propagation compared to single-network or disagreement-only methods.

Experimental results

Research questions

• RQ1 Can two jointly trained networks using cross-updated small-loss samples robustly learn with extremely noisy labels?
• RQ2 How should the small-loss sample rate R(T) be scheduled to balance learning from clean data and avoiding memorization of noise?
• RQ3 Do dual networks offer advantages over single-network or other baselines in heavily corrupted label regimes?

Key findings

• Co-teaching markedly improves robustness over state-of-the-art baselines on noisy MNIST, CIFAR-10, and CIFAR-100 across extreme noise settings.
• Under 45% noisy labels, Co-teaching achieves higher test accuracy than MentorNet and other baselines on MNIST, CIFAR-10, and CIFAR-100.
• Label precision and the ability to isolate clean samples improve for Co-teaching relative to baselines, indicating better identification of clean instances.
• The effectiveness of Co-teaching persists across different noise structures (e.g., Pair flipping and Symmetry flipping) and remains strong when trained from scratch without pre-training.