[Paper Review] Detecting Label Errors in Token Classification Data

Wei-Chen Wang, Jonas Mueller|arXiv (Cornell University)|Oct 8, 2022

Topic Modeling4 citations

TL;DR

This paper proposes worst-token, a method to detect sentences containing label errors in token classification datasets by scoring sentences based on the lowest predicted confidence of any token's label. It evaluates 11 methods using real-world label errors from CoNLL-2003 and finds that worst-token consistently outperforms alternatives in precision-recall, especially when combined with self-confidence scoring, offering a simple, efficient, and effective solution for identifying mislabeled data without requiring model ensembling or complex retraining.

ABSTRACT

Mislabeled examples are a common issue in real-world data, particularly for tasks like token classification where many labels must be chosen on a fine-grained basis. Here we consider the task of finding sentences that contain label errors in token classification datasets. We study 11 different straightforward methods that score tokens/sentences based on the predicted class probabilities output by a (any) token classification model (trained via any procedure). In precision-recall evaluations based on real-world label errors in entity recognition data from CoNLL-2003, we identify a simple and effective method that consistently detects those sentences containing label errors when applied with different token classification models.

Motivation & Objective

• To identify effective methods for detecting sentences containing label errors in token classification datasets.
• To evaluate label error detection performance on real-world, naturally occurring label errors rather than synthetic ones.
• To develop a method that scores entire sentences based on token-level prediction confidence, enabling efficient review of mislabeled examples.
• To compare the effectiveness of confidence-based scoring methods against hard prediction-based approaches in detecting label errors.
• To provide a simple, model-agnostic, and computationally efficient solution for label error detection in NLP datasets.

Proposed method

• The worst-token method scores a sentence based on the minimum predicted probability of the true label across all tokens in the sentence.
• It uses per-token label quality scores derived from model output probabilities: self-confidence (p_ik), normalized margin (p_ik - p_i~k), and confidence-weighted entropy (p_ik / H(p_i)).
• Sentences are ranked by their worst-token score, with lower values indicating higher likelihood of containing at least one mislabeled token.
• The method is model-agnostic and works with any trained token classifier, requiring only out-of-sample predictions.
• Variants such as worst-token-softmin and worst-token-min-alt are explored to improve robustness by considering second-lowest scores or Confident Learning flags.
• Evaluation is performed using precision-recall, AUPRC, and AUROC metrics on real-world CoNLL-2003 entity recognition data with naturally occurring errors.

Experimental results

Research questions

• RQ1 Which label quality scoring method most effectively identifies sentences containing label errors in real-world token classification data?
• RQ2 How does the performance of confidence-based scoring compare to hard prediction-based methods in detecting label errors?
• RQ3 Can a simple, model-agnostic method like worst-token outperform more complex ensemble-based approaches in label error detection?
• RQ4 Does incorporating second-highest or additional confidence scores improve the detection of mislabeled sentences?
• RQ5 How does the performance of label error detection vary across different pre-trained models (e.g., BERT, XLM) and data splits?

Key findings

• The worst-token method with self-confidence scoring achieved the highest Lift@#Errors (9.02) on the BERT model and consistently outperformed other methods across all models and data splits.
• worst-token-softmin achieved slightly better Lift@#Errors than worst-token by considering the second-lowest token confidence, reducing false positives.
• The method achieved AUROC scores above 0.90 across all models, indicating strong discriminative power for identifying error-containing sentences.
• Confidence-weighted entropy (cwe) performed poorly compared to self-confidence and normalized margin, especially in AUPRC and Lift metrics.
• Variants like worst-token-min-alt and worst-token-softmin provided only marginal improvements and introduced higher sensitivity to estimation error, justifying the use of the base worst-token method.
• The study confirms that real-world label errors differ from synthetic ones, and methods trained on synthetic data may not generalize effectively to real datasets.