[Paper Review] CTRL: Clustering Training Losses for Label Error Detection

Chang Yue, Niraj K. Jha|arXiv (Cornell University)|Aug 17, 2022

Machine Learning and Data Classification4 citations

TL;DR

CTRL is a novel framework that detects label errors in multi-class datasets by clustering training loss trajectories of samples using K-means. It identifies noisy labels by exploiting the divergence in learning dynamics between clean and noisy samples, achieving state-of-the-art detection accuracy and enabling a 10%+ improvement in balanced-class accuracy when noise rates are ~20%.

ABSTRACT

In supervised machine learning, use of correct labels is extremely important to ensure high accuracy. Unfortunately, most datasets contain corrupted labels. Machine learning models trained on such datasets do not generalize well. Thus, detecting their label errors can significantly increase their efficacy. We propose a novel framework, called CTRL (Clustering TRaining Losses for label error detection), to detect label errors in multi-class datasets. It detects label errors in two steps based on the observation that models learn clean and noisy labels in different ways. First, we train a neural network using the noisy training dataset and obtain the loss curve for each sample. Then, we apply clustering algorithms to the training losses to group samples into two categories: cleanly-labeled and noisily-labeled. After label error detection, we remove samples with noisy labels and retrain the model. Our experimental results demonstrate state-of-the-art error detection accuracy on both image (CIFAR-10 and CIFAR-100) and tabular datasets under simulated noise. We also use a theoretical analysis to provide insights into why CTRL performs so well.

Motivation & Objective

• Address the pervasive problem of label corruption in supervised machine learning datasets, which undermines model generalization.
• Detect noisy labels in multi-class datasets without requiring ground-truth label information or prior knowledge of noise patterns.
• Leverage the distinct learning dynamics—particularly early memorization of clean labels and delayed fitting of noisy ones—to distinguish between clean and noisy samples.
• Improve model robustness and accuracy by enabling label cleaning through removal of mislabeled instances prior to retraining.
• Provide theoretical justification for the effectiveness of loss-based clustering in separating clean and noisy label learning trajectories.

Proposed method

• Train a neural network on the original noisy dataset to compute per-sample training loss curves over multiple epochs.
• Apply K-means clustering to the loss trajectories of all training samples to group them into two clusters: one corresponding to clean labels and one to noisy labels.
• Use the cluster centroids to assign each sample a label category (clean or noisy) based on its loss trajectory pattern.
• Remove samples classified as noisy and retrain the model on the cleaned dataset to improve generalization and test accuracy.
• The method relies on the observation that models learn clean labels earlier and with lower loss, creating a detectable gap in loss curves before overfitting.
• Theoretical analysis is provided for a binary classification setting, demonstrating that a loss gap between clean and noisy labels is highly probable under standard training dynamics.

Experimental results

Research questions

• RQ1 Can clustering of training loss trajectories effectively distinguish between samples with clean and noisy labels in multi-class datasets?
• RQ2 How does the proposed CTRL framework compare to existing state-of-the-art methods in terms of label error detection accuracy across diverse datasets?
• RQ3 To what extent does removing detected noisy labels improve downstream model performance, particularly in terms of balanced-class accuracy?
• RQ4 What theoretical basis supports the effectiveness of loss-based clustering in identifying label errors?
• RQ5 How robust is the method across different noise levels, noise patterns (e.g., uniform, class-dependent, label-specific), and dataset types (image vs. tabular)?

Key findings

• CTRL achieves state-of-the-art label error detection accuracy on both image and tabular benchmark datasets under various noise settings.
• On datasets with approximately 20% label noise, retraining after CTRL-based cleaning leads to a balanced-class accuracy improvement of over 10%.
• The method maintains comparable or better model test accuracy than prior state-of-the-art methods, even after label cleaning.
• Experiments on tabular datasets (e.g., Credit Fraud, Letter Recognition) show consistent high detection accuracy (e.g., 98.7% mask accuracy at 10% noise) across multiple noise patterns.
• Theoretical analysis confirms that a significant loss gap between clean and noisy labels is highly probable during early training, justifying the method’s design.
• Ablation studies show that removing noisy samples is more effective than alternative label correction or reweighting strategies, especially under high noise rates.