[Paper Review] Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models

Paul Röttger, Haitham Seelawi+3 more|arXiv (Cornell University)|Jun 20, 2022

Hate Speech and Cyberbullying Detection4 citations

TL;DR

Multilingual HateCheck (MHC) introduces a multilingual suite of functional tests for evaluating hate speech detection models across 10 languages—Arabic, Dutch, French, German, Hindi, Italian, Mandarin, Polish, Portuguese, and Spanish—using hand-crafted, culturally adapted test cases. It reveals critical model weaknesses, including over-reliance on keywords, biased target coverage, and inconsistent cross-lingual transfer, even in high-performing multilingual models.

ABSTRACT

Hate speech detection models are typically evaluated on held-out test sets. However, this risks painting an incomplete and potentially misleading picture of model performance because of increasingly well-documented systematic gaps and biases in hate speech datasets. To enable more targeted diagnostic insights, recent research has thus introduced functional tests for hate speech detection models. However, these tests currently only exist for English-language content, which means that they cannot support the development of more effective models in other languages spoken by billions across the world. To help address this issue, we introduce Multilingual HateCheck (MHC), a suite of functional tests for multilingual hate speech detection models. MHC covers 34 functionalities across ten languages, which is more languages than any other hate speech dataset. To illustrate MHC's utility, we train and test a high-performing multilingual hate speech detection model, and reveal critical model weaknesses for monolingual and cross-lingual applications.

Motivation & Objective

• Address the lack of diagnostic evaluation tools for non-English hate speech detection models, which are critical for protecting billions of non-English speakers.
• Overcome the limitations of traditional test-set evaluation, which can mask systematic model biases and over-simplified decision rules.
• Develop a multilingual, functional test suite that enables fine-grained, contrastive evaluation of model capabilities across diverse linguistic and cultural contexts.
• Support the development of more robust, fair, and generalizable multilingual hate speech detection systems by identifying specific failure modes.
• Extend the English HateCheck framework to ten additional languages with native-speaking experts to ensure cultural and linguistic relevance.

Proposed method

• Design 34 functional test categories across 10 languages, each including 25–27 targeted test cases that contrast hateful and non-hateful content.
• Use native-speaking language experts to hand-craft test cases based on the original English HateCheck, adapting them for cultural and linguistic realism in each target language.
• Ensure contrastive design by including non-hateful content that mimics hate speech lexically (e.g., counter-speech or satire) to challenge models relying on keyword matching.
• Apply the MHC suite to evaluate a fine-tuned XLM-T model across monolingual and cross-lingual settings (zero-shot and few-shot) to diagnose model weaknesses.
• Use gold-standard labels from expert-annotated test cases to measure model performance on specific functional capabilities, not just aggregate metrics.
• Make MHC publicly available on GitHub to support reproducibility and community adoption in model development and evaluation.

Experimental results

Research questions

• RQ1 How do multilingual hate speech detection models perform on a diverse, contrastive, and culturally adapted test suite across ten languages?
• RQ2 To what extent do models exhibit over-reliance on keywords and phrases rather than contextual understanding in multilingual settings?
• RQ3 How consistent and reliable is cross-lingual transfer in zero-shot and few-shot settings across different language pairs?
• RQ4 What biases exist in model target coverage, particularly regarding protected groups in different linguistic and cultural contexts?
• RQ5 Can functional testing reveal model weaknesses that are obscured by standard held-out test set evaluations?

Key findings

• The multilingual XLM-T model exhibited significant over-sensitivity to key words and phrases, even when they appeared in non-hateful contexts.
• Model performance was inconsistent across languages, with notable biases in target coverage, particularly for marginalized or stigmatized groups.
• Cross-lingual transfer was error-prone and unreliable in both zero-shot and few-shot settings, indicating poor generalization across languages.
• The model failed to correctly classify non-hateful content that mimicked hate speech (e.g., counter-speech), revealing a lack of contextual understanding.
• Functional testing uncovered model weaknesses that were not apparent from standard held-out test set evaluations, such as inconsistent labeling across similar linguistic patterns.
• MHC successfully identified critical failure modes in a high-performing model, demonstrating its value as a diagnostic tool for multilingual hate speech detection.