[Paper Review] Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification

Daniel Borkan, Lucas Dixon+3 more|arXiv (Cornell University)|Mar 11, 2019

Hate Speech and Cyberbullying Detection12 references4 citations

TL;DR

This paper introduces a suite of threshold-agnostic metrics—based on ROC-AUC, Mann-Whitney U, and Equality Gap—to provide a nuanced assessment of unintended bias in text classification models. Using a newly crowdsourced dataset of nearly 2 million annotated online comments, the authors demonstrate that bias mitigation techniques can reduce bias in short comments but may leave residual bias in longer texts, revealing limitations in current fairness interventions.

ABSTRACT

Unintended bias in Machine Learning can manifest as systemic differences in performance for different demographic groups, potentially compounding existing challenges to fairness in society at large. In this paper, we introduce a suite of threshold-agnostic metrics that provide a nuanced view of this unintended bias, by considering the various ways that a classifier's score distribution can vary across designated groups. We also introduce a large new test set of online comments with crowd-sourced annotations for identity references. We use this to show how our metrics can be used to find new and potentially subtle unintended bias in existing public models.

Motivation & Objective

• To address the limitations of threshold-dependent metrics in measuring unintended bias in text classification.
• To develop a comprehensive, nuanced evaluation framework that captures diverse forms of bias across demographic groups.
• To create and release a large-scale, human-annotated dataset of online comments with identity references for real-world bias evaluation.
• To evaluate the effectiveness of bias mitigation in public toxicity classifiers using real data and new metrics.
• To reveal hidden or subtle biases that are obscured by traditional fairness metrics.

Proposed method

• Propose five threshold-agnostic metrics derived from ROC-AUC, Mann-Whitney U, and Equality Gap to assess model fairness across identity groups.
• Use a synthetic test set and a new large-scale human-annotated dataset of ~2 million comments with identity references to evaluate model performance.
• Apply metrics to two public Perspective API models—one with bias mitigation, one without—to compare bias profiles.
• Visualize score distributions across toxic and non-toxic comments for specific identity groups to illustrate bias patterns.
• Use subgroup AUC, BPSN AUC, and Negative AEG to detect disparities in model behavior across demographic subgroups.
• Analyze the impact of bias mitigation by comparing model behavior on short vs. long comments, especially for identity terms.

Experimental results

Research questions

• RQ1 How do threshold-agnostic metrics reveal different types of unintended bias compared to threshold-dependent metrics?
• RQ2 To what extent does bias mitigation in toxicity classifiers reduce unintended bias across different demographic identity groups?
• RQ3 What patterns of bias persist in real-world text classification models despite known mitigation techniques?
• RQ4 How does comment length influence the manifestation of unintended bias in toxicity classification models?
• RQ5 Can a large-scale, identity-annotated dataset improve the detection of subtle, nuanced bias in text classifiers?

Key findings

• The model with bias mitigation (TOXICITY@6) showed significant improvement in subgroup AUC and BPSN AUC for short comments, particularly for the identity group 'homosexual_gay_or_lesbian'.
• For short comments, the original model (TOXICITY@1) exhibited severe overlap between non-toxic and toxic score distributions for identity-related terms, indicating high bias.
• Despite mitigation, the model still showed higher variance in metrics across identity groups, especially for groups like 'black' and 'asian', which received no additional non-toxic training data.
• The metrics revealed that bias mitigation was most effective in short comments, where training data imbalance was highest, but less effective for longer comments.
• The new dataset enabled detection of previously obscured bias, showing that mitigation did not fully generalize across comment length or identity groups.
• Visualizations confirmed that the mitigation shifted non-toxic scores for identity terms to the left (lower scores), reducing overlap with toxic scores, especially in short comments.