[Paper Review] Assessing the impact of contextual information in hate speech detection

Juan Manuel Cepeda Pérez, Franco M. Luque+9 more|arXiv (Cornell University)|Oct 2, 2022

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This study investigates the impact of contextual information on hate speech detection in Spanish, using a newly annotated corpus of Twitter replies to news articles during the COVID-19 pandemic. Using transformer-based models, the authors demonstrate that incorporating contextual information—particularly the news article headline—improves hate speech detection performance by 4.2 and 5.5 points in Macro F1 for binary and multi-label classification, respectively.

ABSTRACT

In recent years, hate speech has gained great relevance in social networks and other virtual media because of its intensity and its relationship with violent acts against members of protected groups. Due to the great amount of content generated by users, great effort has been made in the research and development of automatic tools to aid the analysis and moderation of this speech, at least in its most threatening forms. One of the limitations of current approaches to automatic hate speech detection is the lack of context. Most studies and resources are performed on data without context; that is, isolated messages without any type of conversational context or the topic being discussed. This restricts the available information to define if a post on a social network is hateful or not. In this work, we provide a novel corpus for contextualized hate speech detection based on user responses to news posts from media outlets on Twitter. This corpus was collected in the Rioplatense dialectal variety of Spanish and focuses on hate speech associated with the COVID-19 pandemic. Classification experiments using state-of-the-art techniques show evidence that adding contextual information improves hate speech detection performance for two proposed tasks (binary and multi-label prediction). We make our code, models, and corpus available for further research.

Motivation & Objective

• Address the limitation of isolated-message hate speech detection by incorporating conversational and topical context.
• Develop a high-quality, manually annotated hate speech dataset in Rioplatense Spanish focused on COVID-19-related discrimination.
• Investigate whether contextual information enhances the performance of state-of-the-art NLP models in hate speech detection tasks.
• Explore multi-label classification beyond binary hate speech detection, including attack targets and calls to action.
• Provide open-source access to the corpus, annotations, and models to support reproducibility and future research.

Proposed method

• Constructed a Twitter-based corpus of replies to news articles from major Argentine media outlets, focusing on COVID-19-related hate speech.
• Annotated a representative sample using detailed, multi-label guidelines covering hate speech, attack targets, and calls to action.
• Employed transformer-based models (e.g., BERT-like architectures) fine-tuned on both isolated comments and context-aware inputs.
• Evaluated performance using Macro F1 scores on binary and multi-label hate speech detection tasks with and without contextual information.
• Compared model performance across different context types, including headlines, full articles, and conversational threads.
• Conducted error analysis to identify persistent challenges, particularly for semantically complex hate speech (e.g., anti-LGBTI rhetoric).

Experimental results

Research questions

• RQ1 Does incorporating contextual information from news articles improve hate speech detection performance in Spanish?
• RQ2 How does the inclusion of context affect the detection of multi-label hate speech features such as attack targets and calls to action?
• RQ3 Which type of contextual information (e.g., headline, full article) yields the best performance in hate speech detection?
• RQ4 To what extent do state-of-the-art models struggle with semantically complex hate speech, such as irony or metaphors targeting the LGBTI community?
• RQ5 Can contextual information mitigate the limitations of isolated-message hate speech detection in low-resource, dialect-specific settings?

Key findings

• Adding contextual information—specifically the news article headline—improved hate speech detection performance by 4.2 points in Macro F1 for binary classification.
• For multi-label classification, the inclusion of context led to a 5.5-point increase in Macro F1, indicating stronger gains for fine-grained hate speech detection.
• The smallest context (the headline) yielded the best performance, suggesting that concise, topic-relevant context is sufficient and effective.
• The model performance did not improve significantly when using longer contexts (e.g., full articles), indicating that excessive context may not add value or could introduce noise.
• The detection of anti-LGBTI hate speech—often involving irony and metaphors—remained challenging, but still benefited significantly from contextual information.
• The study confirms that contextual information is a key factor in improving the robustness and accuracy of hate speech detection systems, especially for complex and subtle forms of discrimination.