[Paper Review] A Novel Self-Supervised Re-labeling Approach for Training with Noisy Labels

Devraj Mandal, Shrisha Bharadwaj+1 more|arXiv (Cornell University)|Oct 13, 2019

Machine Learning and Data Classification35 references4 citations

TL;DR

This paper proposes mCT-S2R, a novel self-supervised re-labeling framework that improves deep learning under noisy labels by first using self-supervision to learn robust features, then leveraging small-loss samples from two parallel networks to re-label high-loss samples. The method significantly outperforms state-of-the-art approaches on MNIST, CIFAR-10, and CIFAR-100 under high noise levels, while reducing computational cost by training with a single network post-relabeling.

ABSTRACT

The major driving force behind the immense success of deep learning models is the availability of large datasets along with their clean labels. Unfortunately, this is very difficult to obtain, which has motivated research on the training of deep models in the presence of label noise and ways to avoid over-fitting on the noisy labels. In this work, we build upon the seminal work in this area, Co-teaching and propose a simple, yet efficient approach termed mCT-S2R (modified co-teaching with self-supervision and re-labeling) for this task. First, to deal with significant amount of noise in the labels, we propose to use self-supervision to generate robust features without using any labels. Next, using a parallel network architecture, an estimate of the clean labeled portion of the data is obtained. Finally, using this data, a portion of the estimated noisy labeled portion is re-labeled, before resuming the network training with the augmented data. Extensive experiments on three standard datasets show the effectiveness of the proposed framework.

Motivation & Objective

• To address the challenge of training deep neural networks when training labels are noisy, a common issue in large-scale datasets from web or crowdsourcing.
• To improve generalization and reduce overfitting to noisy labels without relying on prior knowledge of the noise transition matrix.
• To develop a computationally efficient method that avoids the need for two networks throughout training, unlike co-teaching.
• To enhance model robustness by combining self-supervised pre-training with dynamic re-labeling of high-loss samples using class means from small-loss instances.

Proposed method

• Pre-train the network using a self-supervised contrastive learning objective (e.g., instance discrimination) to learn robust, noise-invariant features without using labels.
• Train two parallel networks in a co-teaching-like setup to identify small-loss samples, which are assumed to be more likely to be correctly labeled.
• Compute class means from the features of the small-loss samples to serve as reference prototypes for re-labeling.
• Re-label high-loss samples based on their similarity to class prototypes, using a confidence threshold (softmax probability) to select only the most reliable re-labeled samples.
• Resume training using a single network with the original small-loss samples and the re-labeled high-loss samples, improving generalization.
• Dynamically adjust the training data distribution using a dropping rate R(T) = 1−ε·min(T/Tk, 1), focusing on clean data as training progresses.

Experimental results

Research questions

• RQ1 Can self-supervised pre-training improve feature robustness and reduce overfitting to noisy labels in image classification?
• RQ2 Can re-labeling high-loss samples using class means derived from small-loss samples improve model accuracy under high noise levels?
• RQ3 Does a two-network co-teaching setup for identifying clean samples, followed by single-network fine-tuning, maintain performance while reducing computational cost?
• RQ4 How robust is the method to variations in hyperparameters such as the confidence threshold (κ) and update frequency (T_update)?
• RQ5 Can the framework generalize across different noise models (e.g., symmetric, pair-20%, pair-45%) and datasets (MNIST, CIFAR-10, CIFAR-100)?

Key findings

• On MNIST with 50% symmetric noise, mCT-S2R achieves 98.86% test accuracy, outperforming the baseline and showing high consistency across hyperparameter settings.
• On CIFAR-100 with 50% symmetric noise, mCT-S2R achieves 60.49% accuracy, significantly outperforming the state-of-the-art under the same noise regime.
• The method maintains strong performance across different confidence thresholds (κ = 0.80, 0.90, 0.95), indicating robustness to hyperparameter choice.
• The framework is insensitive to the choice of which network performs re-labeling, with test accuracy differing by less than 0.3% between the first and second network.
• Even without knowing the true noise level ε, mCT-S2R achieves stable performance (e.g., 94.72% on MNIST with 45% pair noise), suggesting practical usability.
• The method reduces computational cost by transitioning to single-network training after re-labeling, unlike co-teaching which trains two networks throughout.