[Paper Review] Reading Between the Demographic Lines: Resolving Sources of Bias in Toxicity Classifiers

Elizabeth Reichert, Helen Qiu+1 more|arXiv (Cornell University)|Jun 29, 2020

Hate Speech and Cyberbullying Detection12 references4 citations

TL;DR

This paper proposes a fairness-focused approach to reducing identity-driven bias in toxicity classifiers by rebalancing training data using over-sampling, under-sampling, and text generation. It trains logistic regression, neural network, and LSTM models on re-balanced datasets with TF-IDF and GloVe features, achieving a 6.9% false positive rate on identity-related tweets—significantly lower than Perspective API’s 10.3%—demonstrating effective mitigation of bias without sacrificing performance.

ABSTRACT

The censorship of toxic comments is often left to the judgment of imperfect models. Perspective API, a creation of Google technology incubator Jigsaw, is perhaps the most widely used toxicity classifier in industry; the model is employed by several online communities including The New York Times to identify and filter out toxic comments with the goal of preserving online safety. Unfortunately, Google's model tends to unfairly assign higher toxicity scores to comments containing words referring to the identities of commonly targeted groups (e.g., "woman,'' "gay,'' etc.) because these identities are frequently referenced in a disrespectful manner in the training data. As a result, comments generated by marginalized groups referencing their identities are often mistakenly censored. It is important to be cognizant of this unintended bias and strive to mitigate its effects. To address this issue, we have constructed several toxicity classifiers with the intention of reducing unintended bias while maintaining strong classification performance.

Motivation & Objective

• To address the unintended bias in toxicity classifiers that disproportionately flags comments from marginalized groups when they reference their identities.
• To improve fairness in automated toxicity detection while maintaining high classification performance.
• To evaluate whether data rebalancing with synthetic examples can reduce false positives for identity-related content.
• To compare the performance of multiple models (logistic regression, neural networks, LSTM) on re-balanced datasets against Perspective API.
• To analyze the impact of training data composition on model generalization, especially for underrepresented identities.

Proposed method

• Constructed a re-balanced dataset by applying over-sampling, under-sampling, and paraphrasing-based text generation to increase representation of identity-related comments.
• Trained multiple models—logistic regression, feedforward neural networks, and LSTM—using TF-IDF and GloVe embeddings as input features.
• Used a combination of data augmentation techniques to increase the number of toxic and non-toxic examples involving identity terms, aiming for class balance.
• Evaluated models using standard metrics (AUC, F1, FPR, FNR) and compared performance on a held-out test set of identity-related tweets from U.S. House members.
• Benchmarked the best-performing model against Google’s Perspective API on a dataset of 1,984 identity-referencing tweets assumed to be non-toxic.
• Analyzed model misclassifications to identify failure modes, particularly for positive-sentiment tweets involving underrepresented identities like sexuality and religion.

Experimental results

Research questions

• RQ1 Can data rebalancing with synthetic examples reduce false positive rates for identity-related comments in toxicity classifiers?
• RQ2 How does the performance of different model architectures (logistic regression, neural networks, LSTM) compare when trained on re-balanced data?
• RQ3 To what extent does Perspective API over-flag non-toxic identity references compared to a fairness-optimized model?
• RQ4 What role does the diversity and frequency of identity terms in training data play in model generalization and bias mitigation?
• RQ5 Why do models fail to correctly classify positive-sentiment, identity-referencing tweets, and how can this be addressed?

Key findings

• The two-layer neural network model achieved a 6.9% false positive rate on identity-related tweets from U.S. House members, compared to 10.3% for Perspective API, indicating a 33.9% relative reduction in false positives.
• The model maintained high classification performance, with AUC increasing and F1 score remaining stable during incremental data rebalancing.
• False positive rates (FPR) converged and then diverged with increasing toxic identity comments, while false negative rates (FNR) substantially decreased.
• The model incorrectly labeled 78 tweets that Perspective API correctly classified—mostly positive-sentiment messages referencing less common identities like sexuality and religion.
• The model struggled most with identity terms that were underrepresented in the training data, particularly in positive contexts, due to limited data diversity despite text generation.
• Rebalancing the training set to include equal numbers of toxic and non-toxic identity comments significantly improved fairness without degrading overall performance.