[Paper Review] Thou shalt not hate: Countering Online Hate Speech

Binny Mathew, Hardik Tharad+5 more|arXiv (Cornell University)|Jan 1, 2018

Hate Speech and Cyberbullying DetectionComputer Science21 references27 citations

TL;DR

This paper introduces the first large-scale, manually annotated dataset of counterspeech comments from YouTube, enabling the first rigorous linguistic analysis of counterspeech. It proposes machine learning models that detect counterspeech with an F1-score of 0.71 and multilabel types with an F1-score of 0.60, revealing key insights into counterspeech dynamics, effectiveness, and psycholinguistic differences from hate speech.

ABSTRACT

Hate content in social media is ever-increasing. While Facebook, Twitter, Google have attempted to take several steps to tackle the hateful content, they have mostly been unsuccessful. Counterspeech is seen as an effective way of tackling the online hate without any harm to the freedom of speech. Thus, an alternative strategy for these platforms could be to promote counterspeech as a defense against hate content. However, in order to have a successful promotion of such counterspeech, one has to have a deep understanding of its dynamics in the online world. Lack of carefully curated data largely inhibits such understanding. In this paper, we create and release the first ever dataset for counterspeech using comments from YouTube. The data contains 13,924 manually annotated comments where the labels indicate whether a comment is a counterspeech or not. This data allows us to perform a rigorous measurement study characterizing the linguistic structure of counterspeech for the first time. This analysis results in various interesting insights such as: the counterspeech comments receive much more likes as compared to the non-counterspeech comments, for certain communities majority of the non-counterspeech comments tend to be hate speech, the different types of counterspeech are not all equally effective and the language choice of users posting counterspeech is largely different from those posting non-counterspeech as revealed by a detailed psycholinguistic analysis. Finally, we build a set of machine learning models that are able to automatically detect counterspeech in YouTube videos with an F1-score of 0.71. We also build multilabel models that can detect different types of counterspeech in a comment with an F1-score of 0.60.

Motivation & Objective

• To address the growing challenge of online hate speech on social media platforms like YouTube, Facebook, and Twitter.
• To overcome the lack of curated data for studying counterspeech, a non-censoring strategy that counters hate without suppressing free speech.
• To create and release the first large-scale, manually annotated dataset of YouTube comments labeled for counterspeech presence.
• To conduct a comprehensive linguistic and psycholinguistic analysis of counterspeech to understand its structural and behavioral dynamics.
• To develop machine learning models capable of automatically detecting counterspeech and its types in online comments.

Proposed method

• Curated and manually annotated 13,924 YouTube comments to create the first public counterspeech dataset.
• Applied linguistic and psycholinguistic analysis to compare language use in counterspeech versus non-counterspeech comments.
• Trained supervised machine learning models using features derived from lexical, syntactic, and sentiment characteristics of comments.
• Developed both binary classifiers (counterspeech vs. non-counterspeech) and multilabel classifiers (for different types of counterspeech).
• Evaluated models using standard NLP metrics, including F1-score, on the annotated dataset.
• Used statistical analysis to compare engagement (likes) and content patterns between counterspeech and non-counterspeech comments.

Experimental results

Research questions

• RQ1 What are the linguistic and psycholinguistic characteristics that distinguish counterspeech from non-counterspeech comments on YouTube?
• RQ2 How does the engagement (e.g., likes) of counterspeech comments compare to that of non-counterspeech comments?
• RQ3 Which types of counterspeech are most effective in countering hate speech in online communities?
• RQ4 How do the language patterns of users posting counterspeech differ from those posting hate speech?
• RQ5 To what extent can machine learning models accurately detect counterspeech and its subtypes in real-world YouTube comment sections?

Key findings

• Counterspeech comments receive significantly more likes than non-counterspeech comments, indicating higher user engagement and perceived value.
• In certain online communities, a majority of non-counterspeech comments are classified as hate speech, highlighting the prevalence of toxic discourse.
• Different types of counterspeech are not equally effective, suggesting that strategic variation in response style may improve impact.
• Psycholinguistic analysis reveals distinct language use patterns between counterspeech and non-counterspeech posters, particularly in emotional tone and lexical complexity.
• Binary machine learning models achieve an F1-score of 0.71 in detecting counterspeech, demonstrating strong performance on the new dataset.
• Multilabel models for classifying types of counterspeech achieve an F1-score of 0.60, indicating feasible but challenging detection of nuanced response types.