[Paper Review] Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition

Xiaolei Huang, Linzi Xing+2 more|arXiv (Cornell University)|Feb 24, 2020

Hate Speech and Cyberbullying Detection48 references44 citations

TL;DR

This paper presents a multilingual Twitter hate speech corpus with author-level demographic attributes (age, country, gender, race) across English, Italian, Polish, Portuguese, and Spanish, and evaluates baseline classifiers for performance and fairness while analyzing language-demographic variation.

ABSTRACT

Existing research on fairness evaluation of document classification models mainly uses synthetic monolingual data without ground truth for author demographic attributes. In this work, we assemble and publish a multilingual Twitter corpus for the task of hate speech detection with inferred four author demographic factors: age, country, gender and race/ethnicity. The corpus covers five languages: English, Italian, Polish, Portuguese and Spanish. We evaluate the inferred demographic labels with a crowdsourcing platform, Figure Eight. To examine factors that can cause biases, we take an empirical analysis of demographic predictability on the English corpus. We measure the performance of four popular document classifiers and evaluate the fairness and bias of the baseline classifiers on the author-level demographic attributes.

Motivation & Objective

• Create a multilingual hate speech corpus annotated with author-level demographic attributes to enable fairness evaluation.
• Incorporate demographic attributes inferred from public profiles (age, country, gender, race) to study bias in hate speech detection.
• Provide baseline classification results across languages and analyze fairness using group-based metrics.

Proposed method

• Merge and binarize hate speech labels from five language datasets to form a unified multilingual corpus.
• Infer demographic attributes from profile images (Face++) and location data (Google Maps) while excluding tweet content for attribute inference.
• Assess demographic attribute inference accuracy via crowdsourcing on a sample of profiles.
• Evaluate four baseline classifiers (Logistic Regression, CNN, RNN, BERT) on hate speech detection across languages.
• Measure fairness using equality differences for true/false positive/negative rates across demographic groups.

Experimental results

Research questions

• RQ1 How do author-level demographic attributes influence hate speech detection performance across multiple languages?
• RQ2 What is the fairness profile of common document classifiers when evaluated on demographic groups in a multilingual hate speech corpus?
• RQ3 How predictable are demographic attributes from text versus profile-derived features, and what does this imply for fairness analyses?

Key findings

• Baseline classifiers show varying performance across languages; neural models (CNN, RNN) generally outperform logistic regression, while BERT underperforms in some languages due to domain generalization issues.
• Age and gender exhibit more pronounced fairness differences in several languages, with Polish and Italian showing stronger biases for age/gender; country- and race-related biases vary by language.
• English data tends to be the most unbiased across attributes, while Polish/Italian datasets show higher bias for age and gender in several models.
• There is no strong, consistent correlation between overall classification performance and fairness bias across languages and models.