[Paper Review] Spread Love Not Hate: Undermining the Importance of Hateful Pre-training for Hate Speech Detection

Omkar Gokhale, Aditya Kane+3 more|arXiv (Cornell University)|Oct 9, 2022

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This paper challenges the assumption that hateful pre-training is essential for hate speech detection by showing that models pre-trained on non-hateful or random in-domain tweets outperform or match those pre-trained on hateful content in Hindi and Marathi. It introduces MahaTweetBERT and HindTweetBERT—state-of-the-art BERT models for Indian languages—and releases them with gold-standard hate speech benchmarks, demonstrating that domain adaptation, not hate-specific pre-training, drives performance gains.

ABSTRACT

Pre-training large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. Although this method has proven to be effective for many domains, it might not always provide desirable benefits. In this paper, we study the effects of hateful pre-training on low-resource hate speech classification tasks. While previous studies on the English language have emphasized its importance, we aim to augment their observations with some non-obvious insights. We evaluate different variations of tweet-based BERT models pre-trained on hateful, non-hateful, and mixed subsets of a 40M tweet dataset. This evaluation is carried out for the Indian languages Hindi and Marathi. This paper is empirical evidence that hateful pre-training is not the best pre-training option for hate speech detection. We show that pre-training on non-hateful text from the target domain provides similar or better results. Further, we introduce HindTweetBERT and MahaTweetBERT, the first publicly available BERT models pre-trained on Hindi and Marathi tweets, respectively. We show that they provide state-of-the-art performance on hate speech classification tasks. We also release hateful BERT for the two languages and a gold hate speech evaluation benchmark HateEval-Hi and HateEval-Mr consisting of manually labeled 2000 tweets each. The models and data are available at https://github.com/l3cube-pune/MarathiNLP .

Motivation & Objective

• To investigate whether hateful pre-training is truly necessary for effective hate speech detection in low-resource Indian languages like Hindi and Marathi.
• To evaluate the performance of BERT models pre-trained on hateful, non-hateful, and random subsets of a 40M tweet corpus for hate speech classification.
• To develop and release the first publicly available BERT models—MahaTweetBERT and HindTweetBERT—pre-trained on Marathi and Hindi tweets, respectively.
• To create gold-standard hate speech evaluation benchmarks, HateEval-Hi and HateEval-Mr, with 2,000 manually labeled tweets per language.
• To demonstrate that domain-specific adaptation through in-domain pre-training, regardless of content polarity, yields superior performance over hate-specific pre-training.

Proposed method

• Pre-trained MahaBERT and HindBERT models on 40 million Marathi and Hindi tweets, respectively, using the masked language modeling (MLM) objective.
• Extracted three subsets from the 40M tweet corpus: most hateful (top 1M), least hateful (bottom 1M), and random (1M) tweets using confidence scores from prior state-of-the-art hate speech models.
• Fine-tuned each model variant on downstream hate speech detection datasets (HASOC and CONSTRAINT) using a 25-epoch training schedule with a 5e-6 initial learning rate.
• Used Hugging Face’s transformers library for training and model hosting, ensuring reproducibility and accessibility.
• Conducted five random seed runs per experiment and reported mean macro-F1 scores to ensure statistical reliability.
• Released all models, datasets, and training code on GitHub under the L3Cube-MarathiNLP project for community use.

Experimental results

Research questions

• RQ1 Does pre-training on hateful tweets lead to superior performance in hate speech detection for low-resource Indic languages like Hindi and Marathi?
• RQ2 Is the performance gain from hateful pre-training due to domain adaptation or the specific hateful content of the pre-training data?
• RQ3 How do monolingual models pre-trained on in-domain tweets compare to multilingual models like MuRIL in hate speech detection tasks?
• RQ4 Can models pre-trained on non-hateful or random in-domain text achieve performance comparable to or better than those pre-trained on hateful content?
• RQ5 To what extent does large-scale pre-training on 40 million tweets improve downstream hate speech classification beyond smaller, curated subsets?

Key findings

• Models pre-trained on non-hateful or random subsets of tweets outperformed or matched those pre-trained on hateful content in hate speech detection tasks for both Hindi and Marathi.
• The MahaTweetBERT and HindTweetBERT models, pre-trained on 40 million in-domain tweets, achieved state-of-the-art performance across all downstream datasets, outperforming even the hateful variants.
• The hateful pre-training variant (e.g., MahaTweetBERT-Hateful) did not yield the best results; in fact, it underperformed models trained on non-hateful or random data on two out of three Marathi datasets and both Hindi datasets in terms of macro-F1.
• Monolingual models retrained on large in-domain tweet corpora significantly outperformed the multilingual MuRIL model, which was pre-trained on 17 Indian languages and billions of tokens.
• The performance gain from hateful pre-training is likely a side effect of domain adaptation rather than a direct result of learning hate-specific linguistic patterns.
• The release of HateEval-Hi and HateEval-Mr, each with 2,000 manually annotated tweets, provides the first gold-standard benchmarks for hate speech detection in Hindi and Marathi.