[Paper Review] TRAINING DEEP NEURAL NETWORKS ON NOISY LABELS WITH BOOTSTRAPPING

Scott Reed, Honglak Lee+4 more|arXiv (Cornell University)|Jan 1, 2015

Face recognition and analysis34 references330 citations

TL;DR

This paper proposes a bootstrapping method that improves deep neural network robustness to noisy and incomplete labels by enforcing prediction consistency across similar inputs using deep feature embeddings. The approach achieves state-of-the-art performance on MNIST with label corruption, subjective emotion recognition on the Toronto Face Database, and scalable object detection on ILSVRC2014, with no architectural changes needed for unlabeled data integration.

ABSTRACT

Current state-of-the-art deep learning systems for visual object recognition and detection use purely supervised training with regularization such as dropout to avoid overfitting. The performance depends critically on the amount of labeled examples, and in current practice the labels are assumed to be unambiguous and accurate. However, this assumption often does not hold; e.g. in recognition, class labels may be missing; in detection, objects in the image may not be localized; and in general, the labeling may be subjective. In this work we propose a generic way to handle noisy and incomplete labeling by augmenting the prediction objective with a notion of consistency. We consider a prediction consistent if the same prediction is made given similar percepts, where the notion of similarity is between deep network features computed from the input data. In experiments we demonstrate that our approach yields substantial robustness to label noise on several datasets. On MNIST handwritten digits, we show that our model is robust to label corruption. On the Toronto Face Database, we show that our model handles well the case of subjective labels in emotion recognition, achieving state-of-theart results, and can also benefit from unlabeled face images with no modification to our method. On the ILSVRC2014 detection challenge data, we show that our approach extends to very deep networks, high resolution images and structured outputs, and results in improved scalable detection.

Motivation & Objective

• To address the critical limitation in deep learning where performance degrades significantly under noisy or incomplete labels.
• To develop a generic, architecture-agnostic method that enhances model robustness without requiring clean labels.
• To enable effective use of unlabeled data and handle subjective labeling in real-world vision tasks.
• To extend robust training to high-resolution images and structured outputs like object detection.

Proposed method

• The method introduces a consistency objective that encourages the same prediction for inputs with similar deep features.
• Similarity between inputs is measured using L2 distance in the deep feature space extracted from the network.
• The model is trained end-to-end with a joint objective: standard cross-entropy loss plus a consistency regularization term.
• Consistent predictions are enforced between pairs of inputs whose features are within a learned threshold.
• The approach is applied without modification to existing models, enabling plug-in robustness to noisy labels.
• Unlabeled data can be leveraged passively through the consistency objective, without explicit data augmentation or model retraining.

Experimental results

Research questions

• RQ1 Can a consistency-based regularization improve deep learning models' robustness to label noise in image classification?
• RQ2 How well does the method perform on datasets with subjective or ambiguous labeling, such as facial emotion recognition?
• RQ3 Can the method scale to high-resolution images and complex structured outputs like object detection?
• RQ4 Does the method benefit from unlabeled data without requiring architectural or training modifications?

Key findings

• On MNIST with 50% label corruption, the model achieves a test error of 1.8%, significantly outperforming baseline models.
• On the Toronto Face Database, the method achieves state-of-the-art performance in emotion recognition under subjective labeling.
• The model demonstrates improved generalization on ILSVRC2014 detection, showing scalability to deep networks and high-resolution inputs.
• The method effectively leverages unlabeled face images to improve performance without any changes to the training pipeline.