[Paper Review] AQuA: A Benchmarking Tool for Label Quality Assessment

Mononito Goswami, Vedant Sanil+4 more|arXiv (Cornell University)|Jun 15, 2023

Machine Learning and Data Classification4 citations

TL;DR

AQuA is a comprehensive benchmarking framework designed to rigorously evaluate label quality assessment methods in machine learning under label noise. It standardizes evaluation across diverse datasets, noise types, and models, revealing that Confident Learning removes excessive data, while SimiFeat and AUM show superior robustness and performance across modalities and noise settings.

ABSTRACT

Machine learning (ML) models are only as good as the data they are trained on. But recent studies have found datasets widely used to train and evaluate ML models, e.g. ImageNet, to have pervasive labeling errors. Erroneous labels on the train set hurt ML models' ability to generalize, and they impact evaluation and model selection using the test set. Consequently, learning in the presence of labeling errors is an active area of research, yet this field lacks a comprehensive benchmark to evaluate these methods. Most of these methods are evaluated on a few computer vision datasets with significant variance in the experimental protocols. With such a large pool of methods and inconsistent evaluation, it is also unclear how ML practitioners can choose the right models to assess label quality in their data. To this end, we propose a benchmarking environment AQuA to rigorously evaluate methods that enable machine learning in the presence of label noise. We also introduce a design space to delineate concrete design choices of label error detection models. We hope that our proposed design space and benchmark enable practitioners to choose the right tools to improve their label quality and that our benchmark enables objective and rigorous evaluation of machine learning tools facing mislabeled data.

Motivation & Objective

• To address the lack of standardized evaluation for label noise detection methods in machine learning.
• To provide a unified benchmarking framework that enables objective, reproducible, and comprehensive evaluation of label quality assessment tools.
• To identify the most effective label cleaning methods across diverse data modalities, noise types, and model architectures.
• To define a design space for label error detection models to guide practitioners in selecting appropriate tools for their data.
• To reveal empirical insights into the performance and robustness of existing label cleaning methods under controlled, large-scale experiments.

Proposed method

• AQuA integrates 4 data modalities (image, tabular, text, time-series), 4 single-label and 3 multi-annotator noise injection methods, and 4 state-of-the-art label error detection models.
• It employs 10+ classification models (e.g., ResNet-18, Distil-RoBERTa, MLP) across all experiments to assess downstream performance.
• The framework uses multiple evaluation metrics beyond accuracy, including F1-score, AUC-ROC, fairness, generalization, and robustness, to assess model behavior.
• Noise injection includes uniform, asymmetric, class-dependent, and instance-dependent types at varying noise rates (0.01–0.5) across 1000+ unique experimental configurations.
• AQuA enables segmentation of results by data modality, noise type, and model architecture to identify consistent performance trends.
• The benchmark includes full reproducibility via datacards and open-source code on GitHub, ensuring transparency and reusability.

Experimental results

Research questions

• RQ1 Which label error detection methods perform most consistently across diverse data modalities and noise types?
• RQ2 How does the choice of base classification model affect the performance of label cleaning methods?
• RQ3 To what extent do different noise injection strategies (e.g., uniform vs. asymmetric) impact the detectability of labeling errors?
• RQ4 How do label cleaning methods affect downstream model performance, fairness, and robustness?
• RQ5 What is the impact of data imbalance and over-cleaning on model generalization and performance?

Key findings

• Confident Learning removes on average 57% of training data points, significantly reducing model capacity and leading to poor downstream performance despite high error detection accuracy.
• SimiFeat and AUM consistently rank among the top methods in both label error detection accuracy and downstream model F1-score across all data modalities.
• Cleanlab performs best in identifying labeling errors when measured by weighted F1, especially in image and tabular data, but shows lower performance in time-series and text data.
• The performance of label cleaning methods varies significantly by noise type: asymmetric and instance-dependent noise are more challenging to detect than uniform noise.
• CINCER and AUM show minimal impact on class imbalance, while Confident Learning significantly alters minority class representation, potentially harming fairness.
• Across all experiments, SimiFeat and AUM demonstrate superior robustness, with consistent performance across noise rates, data types, and downstream models.