[Paper Review] MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels

Lu Jiang, Zhengyuan Zhou+3 more|arXiv (Cornell University)|Dec 14, 2017

Human Pose and Action Recognition31 references536 citations

TL;DR

MentorNet learns a data-driven curriculum to supervise StudentNet training, improving generalization for very deep networks trained on corrupted or noisy labels, including large-scale WebVision and ImageNet experiments.

ABSTRACT

Recent deep networks are capable of memorizing the entire data even when the labels are completely random. To overcome the overfitting on corrupted labels, we propose a novel technique of learning another neural network, called MentorNet, to supervise the training of the base deep networks, namely, StudentNet. During training, MentorNet provides a curriculum (sample weighting scheme) for StudentNet to focus on the sample the label of which is probably correct. Unlike the existing curriculum that is usually predefined by human experts, MentorNet learns a data-driven curriculum dynamically with StudentNet. Experimental results demonstrate that our approach can significantly improve the generalization performance of deep networks trained on corrupted training data. Notably, to the best of our knowledge, we achieve the best-published result on WebVision, a large benchmark containing 2.2 million images of real-world noisy labels. The code are at https://github.com/google/mentornet

Motivation & Objective

• Motivate the problem of deep nets memorizing corrupted labels and the need for robust generalization.
• Propose a data-driven curriculum learning approach to supervise backbone networks during training.
• Demonstrate scalability of the method to large deep architectures and big datasets.
• Show empirical gains over predefined curriculums on both controlled and real-world noisy labels.

Proposed method

• Introduce MentorNet to learn sample weights that form a curriculum by observing StudentNet feedback.
• Formulate a joint objective with StudentNet where MentorNet outputs per-sample weights via a learned function g_m(z_i;Θ).
• Allow MentorNet to approximate predefined curricula or discover data-driven curricula from data.
• Use SPADE (Scholastic gradient Partial DEScent) to optimize StudentNet jointly with MentorNet in mini-batch SGD, enabling parallel training.
• Provide a burn-in period and a dynamic curriculum that updates MentorNet during training.
• Connect the learned curriculum to robust objective interpretations via M-estimators (Huber, log-sum penalties).

Experimental results

Research questions

• RQ1 Can a learned, data-driven curriculum improve generalization of very deep networks trained on corrupted labels?
• RQ2 How does a jointly learned MentorNet–StudentNet framework compare to predefined curricula on controlled and real-world noisy labels?
• RQ3 Is the proposed training algorithm scalable and convergent for large-scale datasets and architectures?
• RQ4 What underlying robust objective does the learned curriculum imply for training with label noise?

Key findings

• MentorNet with data-driven curricula significantly improves validation accuracy over baseline training on controlled corrupted-label benchmarks.
• The learned curriculum (MentorNet DD) outperforms predefined curriculums such as Self-paced and Focal Loss in experiments.
• On large-scale ImageNet with 40% noisy labels, MentorNet improves P@1 and P@5 over several baselines, including no-regularization and fully regularized models.
• On WebVision with real-world noisy labels, MentorNet substantially improves baseline methods when evaluated on clean ImageNet and WebVision validations.
• MentorNet demonstrates convergence and stable training, with training error approaching zero while test performance remains robust to label noise.