[Paper Review] A Benchmark Dataset for Learning to Intervene in Online Hate Speech

Jing Qian, Anna Bethke+3 more|arXiv (Cornell University)|Sep 10, 2019

Hate Speech and Cyberbullying Detection32 references7 citations

TL;DR

This paper introduces a novel task of generative hate speech intervention, proposing two large-scale, fully-labeled datasets from Reddit and Gab that include conversational context and human-written intervention responses. The key contribution is a benchmark for training and evaluating models to generate context-aware, effective responses that counter hate speech, with findings showing significant room for improvement over automatic methods compared to human-written interventions.

ABSTRACT

Countering online hate speech is a critical yet challenging task, but one which can be aided by the use of Natural Language Processing (NLP) techniques. Previous research has primarily focused on the development of NLP methods to automatically and effectively detect online hate speech while disregarding further action needed to calm and discourage individuals from using hate speech in the future. In addition, most existing hate speech datasets treat each post as an isolated instance, ignoring the conversational context. In this paper, we propose a novel task of generative hate speech intervention, where the goal is to automatically generate responses to intervene during online conversations that contain hate speech. As a part of this work, we introduce two fully-labeled large-scale hate speech intervention datasets collected from Gab and Reddit. These datasets provide conversation segments, hate speech labels, as well as intervention responses written by Mechanical Turk Workers. In this paper, we also analyze the datasets to understand the common intervention strategies and explore the performance of common automatic response generation methods on these new datasets to provide a benchmark for future research.

Motivation & Objective

• To address the limitations of existing hate speech detection by proposing a generative intervention task that goes beyond detection to actively mitigate hate speech through response generation.
• To create two large-scale, fully-labeled datasets from Reddit and Gab that preserve conversational context and include human-written intervention responses.
• To provide a benchmark for evaluating automatic response generation models in the context of online hate speech intervention.
• To analyze common intervention strategies used by humans and evaluate the performance of state-of-the-art generative models on these datasets.

Proposed method

• The datasets were collected from Reddit and Gab, with conversations segmented and labeled for hate speech using Mechanical Turk workers.
• Each conversation includes a hateful post, the preceding context, and a human-written intervention response, ensuring contextual awareness.
• The study evaluates multiple generative models: Sequence-to-Sequence (Seq2Seq), Variational Autoencoder (VAE), and Reinforcement Learning (RL)-fine-tuned models.
• Automatic evaluation metrics include BLEU, ROUGE, and METEOR, while human evaluation assesses effectiveness and diversity of responses.
• Models are trained and tested on both datasets, with input filtering applied to improve context relevance.
• The study compares model performance using both automatic metrics and human annotation to assess real-world effectiveness.

Experimental results

Research questions

• RQ1 What are the most common intervention strategies used by humans in response to online hate speech?
• RQ2 How do automatic response generation models perform on context-aware hate speech intervention compared to human-written responses?
• RQ3 To what extent do automatic evaluation metrics like BLEU, ROUGE, and METEOR correlate with human judgment of response quality?
• RQ4 Why do certain models, such as VAE, generate less diverse responses despite high automatic scores?
• RQ5 How does dataset size and distribution (e.g., Gab vs. Reddit) affect model generalization and performance?

Key findings

• The VAE model generated responses with significantly lower diversity in human evaluation, often repeating generic phrases related to specific hate keywords, such as those targeting disabilities.
• Despite high automatic scores (e.g., 4.2 BLEU, 20.4 ROUGE, 18.2 METEOR), a universal response like 'Please do not use derogatory language' achieved strong metric scores but was judged low in quality due to lack of context.
• Reinforcement Learning (RL)-fine-tuned models produced responses rated as more effective and diverse by human annotators, especially on the Gab dataset, despite not outperforming others on automatic metrics.
• The Seq2Seq model with filtered conversation input achieved higher automatic scores, indicating that context filtering improves performance.
• Human-written responses were preferred over machine-generated ones in 70% or more of human evaluations on both datasets, highlighting the gap in quality and contextual relevance.
• The Reddit dataset, being smaller and noisier due to longer average post lengths, proved more challenging for models, especially with limited training data.