[Paper Review] DivideMix: Learning with Noisy Labels as Semi-supervised Learning

Junnan Li, Richard Socher+1 more|arXiv (Cornell University)|Feb 18, 2020

Machine Learning and Data ClassificationComputer Science44 references499 citations

TL;DR

DivideMix treats learning with noisy labels as a semi-supervised problem by co-dividing data using per-sample loss modeled by a Gaussian Mixture Model, and enhances semi-supervised learning with label refinement and co-guessing across two diverged networks.

ABSTRACT

Deep neural networks are known to be annotation-hungry. Numerous efforts have been devoted to reducing the annotation cost when learning with deep networks. Two prominent directions include learning with noisy labels and semi-supervised learning by exploiting unlabeled data. In this work, we propose DivideMix, a novel framework for learning with noisy labels by leveraging semi-supervised learning techniques. In particular, DivideMix models the per-sample loss distribution with a mixture model to dynamically divide the training data into a labeled set with clean samples and an unlabeled set with noisy samples, and trains the model on both the labeled and unlabeled data in a semi-supervised manner. To avoid confirmation bias, we simultaneously train two diverged networks where each network uses the dataset division from the other network. During the semi-supervised training phase, we improve the MixMatch strategy by performing label co-refinement and label co-guessing on labeled and unlabeled samples, respectively. Experiments on multiple benchmark datasets demonstrate substantial improvements over state-of-the-art methods. Code is available at https://github.com/LiJunnan1992/DivideMix .

Motivation & Objective

• Address the challenge of training deep networks with noisy labels while reducing annotation cost.
• Leverage unlabeled/noisy data through semi-supervised learning to improve generalization.
• Develop a robust mechanism to separate clean and noisy samples during training.
• Mitigate confirmation bias by using two diverged networks that teach each other.
• Improve SSL through label refinement and co-guessing tailored for noisy labels.

Proposed method

• Model per-sample loss with a Gaussian Mixture Model to estimate clean/noisy probabilities for each sample.
• Use co-divide: two networks alternately divide data into labeled (clean) and unlabeled (noisy) sets for the other network.
• Warm-up with confidence penalty to stabilize learning on asymmetric noise.
• Apply an enhanced MixMatch SSL framework with label co-refinement (refining labeled sample labels using predicted distributions) and co-guessing (ensemble predictions for unlabeled data).
• Perform data augmentation and MixUp-based mixing of labeled and unlabeled samples, with a regularization term enforcing a uniform prior across classes.
• Update networks via SGD using a combined loss: supervised on labeled data, unsupervised on unlabeled data, and a regularization term.

Experimental results

Research questions

• RQ1 Can noisy labeled data be effectively utilized by reframing the task as semi-supervised learning?
• RQ2 Does co-training two divergent networks reduce confirmation bias and improve robustness to label noise?
• RQ3 How can per-sample loss modeling and MixMatch be adapted to handle asymmetric and high-noise regimes?
• RQ4 What is the impact of label refinement and co-guessing on semi-supervised performance with noisy labels?
• RQ5 To what extent does DivideMix outperform existing LNL and SSL methods across benchmarks?

Key findings

• DivideMix achieves substantial improvements over state-of-the-art methods across CIFAR-10/100 with symmetric and asymmetric noise.
• On CIFAR-10 with 40% asymmetric noise, DivideMix attains a top performance superior to competing methods.
• DivideMix consistently outperforms baselines on Clothing1M and WebVision benchmarks.
• Ablation studies show that co-divide, label refinement, and co-guessing contribute to performance gains and that two diverged networks help avoid confirmation bias.
• Ensemble predictions from two networks during inference yield better results than a single model.