[Paper Review] Training Convolutional Networks with Noisy Labels

Sainbayar Sukhbaatar, Joan Bruna+3 more|arXiv (Cornell University)|Jun 9, 2014

Machine Learning and Data Classification31 references515 citations

TL;DR

This paper proposes a noise-adaptive layer for Convolutional Networks that improves robustness to noisy labels by modeling label flips and outliers. By adding a constrained linear noise layer on top of the softmax, the model learns to correct for label distribution shifts during end-to-end training, achieving significant accuracy gains on synthetic and real-world noisy datasets, including ImageNet with up to 1.2M outlier images.

ABSTRACT

The availability of large labeled datasets has allowed Convolutional Network models to achieve impressive recognition results. However, in many settings manual annotation of the data is impractical; instead our data has noisy labels, i.e. there is some freely available label for each image which may or may not be accurate. In this paper, we explore the performance of discriminatively-trained Convnets when trained on such noisy data. We introduce an extra noise layer into the network which adapts the network outputs to match the noisy label distribution. The parameters of this noise layer can be estimated as part of the training process and involve simple modifications to current training infrastructures for deep networks. We demonstrate the approaches on several datasets, including large scale experiments on the ImageNet classification benchmark.

Motivation & Objective

• To investigate the robustness of standard ConvNets when trained on datasets with noisy labels, particularly in real-world settings where labels are unreliable.
• To address the challenge of training deep networks when label noise is prevalent, such as in web-scraped or crowd-sourced datasets.
• To propose a simple, trainable noise layer that adapts network outputs to match the observed noisy label distribution without requiring clean labels.
• To evaluate the effectiveness of the method on both synthetic and real-world noisy data, including ImageNet with high levels of outlier noise.

Proposed method

• Introduces a constrained linear noise layer on top of the softmax layer to model label noise, which is trained end-to-end with backpropagation.
• For label flip noise, the method uses a learnable noise matrix Q that models the probability of mislabeling between classes.
• For outlier noise, a separate noise layer models the probability of an image being an outlier and assigned a random class label.
• The noise layer parameters are estimated during training using standard optimization, requiring minimal modification to existing deep learning frameworks.
• The method uses a hyperparameter α to control the prior probability of outliers, which can be tuned or estimated from data.
• The approach is scalable and applied to large-scale datasets like ImageNet with up to 1.2M outlier images.

Experimental results

Research questions

• RQ1 How does standard ConvNet performance degrade under high levels of label noise, particularly label flips and outliers?
• RQ2 Can a simple, differentiable noise layer improve model robustness to both label flip and outlier noise in deep networks?
• RQ3 How sensitive is the performance of the noise-adapted model to the choice of hyperparameter α, especially when the true outlier fraction is unknown?
• RQ4 Does the noise-adaptive layer provide measurable gains on real-world noisy datasets, such as web-scraped images or Tiny Images?

Key findings

• On ImageNet with 1.2M outlier images (60% inliers), the noise-adapted model reduced top-1 error by 1.5% compared to the standard ConvNet.
• On the Tiny Images dataset with an estimated 50% outlier fraction, the noise-adapted model reduced test error from 19.2% to 18.8%, a 2.1% relative improvement.
• On a real-world web image dataset with 0.9M images and 1,000 ImageNet classes, the label-flip noise model reduced validation error from 48.8% to 48.2%.
• The method remained robust even when α was perturbed by ±15% from the true outlier fraction, indicating low sensitivity to hyperparameter choice.
• The noise layer achieved significant gains on synthetic data with high noise levels, demonstrating its effectiveness in controlled settings.
• The approach is easily implementable in existing deep learning frameworks and scales to large-scale problems like ImageNet without major architectural changes.