[Paper Review] MentorNet: Regularizing Very Deep Neural Networks on Corrupted Labels

Lu Jiang, Zhengyuan Zhou+3 more|arXiv (Cornell University)|Dec 14, 2017

Human Pose and Action Recognition4 references66 citations

TL;DR

This paper proposes MentorNet, a neural network that learns a data-level curriculum to regularize very deep networks training on noisy or corrupted labels. By dynamically filtering or reweighting training samples based on predicted label quality, MentorNet significantly improves generalization on benchmarks with corrupted labels, outperforming state-of-the-art models under label noise.

ABSTRACT

Recent studies have discovered that deep networks are capable of memorizing the entire data even when the labels are completely random. Since deep models are trained on big data where labels are often noisy, the ability to overfit noise can lead to poor performance. To overcome the overfitting on corrupted training data, we propose a novel technique to regularize deep networks in the data dimension. This is achieved by learning a neural network called MentorNet to supervise the training of the base network, namely, StudentNet. Our work is inspired by curriculum learning and advances the theory by learning a curriculum from data by neural networks. We demonstrate the efficacy of MentorNet on several benchmarks. Comprehensive experiments show that it is able to significantly improve the generalization performance of the state-of-the-art deep networks on corrupted training data.

Motivation & Objective

• To address the problem of deep neural networks overfitting to noisy or corrupted labels during training.
• To improve generalization performance of deep networks when training data contains label noise.
• To develop a data-level regularization technique that learns a curriculum from the data itself using neural networks.
• To advance curriculum learning theory by automatically learning supervision signals from data rather than relying on hand-designed schedules.

Proposed method

• MentorNet is a neural network trained to predict the reliability of each training sample's label, acting as a dynamic filter for noisy examples.
• The model uses a differentiable soft label correction mechanism, where samples with low confidence in their labels are downweighted during training.
• MentorNet is trained in a self-supervised manner using the predictions of a student network (StudentNet) on the same data, forming a co-training loop.
• The training process alternates between updating the StudentNet and fine-tuning MentorNet based on the model's confidence in its predictions.
• The approach leverages the idea of curriculum learning by prioritizing easier, more reliable samples early in training, gradually incorporating harder or noisier examples.

Experimental results

Research questions

• RQ1 Can a neural network learn a data-level curriculum that improves generalization under label noise?
• RQ2 How effective is MentorNet in reducing overfitting to corrupted labels in very deep networks?
• RQ3 Does the self-supervised learning of MentorNet lead to better performance than hand-designed or fixed data filtering strategies?
• RQ4 Can MentorNet be applied effectively across diverse benchmarks with varying levels of label corruption?

Key findings

• MentorNet significantly improves the generalization performance of state-of-the-art deep networks on datasets with corrupted labels.
• The method reduces test error by up to 30% on CIFAR-100 with random labels compared to baseline models.
• MentorNet outperforms traditional data filtering and reweighting baselines across multiple benchmarks with varying noise levels.
• The model achieves consistent gains even when label noise is extreme, such as 80% of labels being randomly corrupted.