[Paper Review] Analyzing the hate and counter speech accounts on Twitter

Binny Mathew, N. Ravi Kumar+3 more|arXiv (Cornell University)|Dec 6, 2018

Hate Speech and Cyberbullying DetectionComputer Science32 references58 citations

TL;DR

The paper builds a dataset of hate tweets and counterspeech replies on Twitter, analyzes linguistic and psycholinguistic traits, profiles hate and counter accounts, and trains a classifier that achieves 0.78 F1 (78% accuracy) to distinguish hate from counter accounts.

ABSTRACT

The online hate speech is proliferating with several organization and countries implementing laws to ban such harmful speech. While these restrictions might reduce the amount of such hateful content, it does so by restricting freedom of speech. Thus, an promising alternative supported by several organizations is to counter such hate speech with more speech. In this paper, We analyze hate speech and the corresponding counters (aka counterspeech) on Twitter. We perform several lexical, linguistic and psycholinguistic analysis on these user accounts and obverse that counter speakers employ several strategies depending on the target community. The hateful accounts express more negative sentiments and are more profane. We also find that the hate tweets by verified accounts have much more virality as compared to a tweet by a non-verified account. While the hate users seem to use words more about envy, hate, negative emotion, swearing terms, ugliness, the counter users use more words related to government, law, leader. We also build a supervised model for classifying the hateful and counterspeech accounts on Twitter and obtain an F-score of 0.77. We also make our dataset public to help advance the research on hate speech.

Motivation & Objective

• Motivate and study counterspeech as an alternative to blocking hatespeech on Twitter.
• Create a dataset of hate tweets and their counterspeech replies for analysis and modeling.
• Characterize hate and counterspeech accounts across activity, lexical, personality, and topical dimensions.
• Develop a predictive model to automatically distinguish hateful from counterspeech accounts.
• Provide insights on how counterspeech strategies vary by target community and platform dynamics.

Proposed method

• Curate a dataset of 1290 counterspeech replies to 558 hate tweets, yielding 1290 counterspeech replies from 1239 accounts and 558 hate tweets from 548 hate accounts.
• Annotate tweets to identify hate content and classify counterspeech into predefined categories (with inter-annotator agreement metrics).
• Extract and analyze lexical, sentiment, profanity, and psycholinguistic features (e.g., Empath categories, IBM Watson personality traits).
• Construct user-level features from 3200-tweet histories (TF-IDF, profile metrics, lexical/affect features) for each account.
• Train and evaluate multiple classifiers (SVM, LR, RF, ET, XGBoost, CatBoost) to distinguish hate vs counterspeech accounts, selecting CatBoost as best.
• Perform feature ablation to assess the contribution of TF-IDF, lexical, and affect features.

Experimental results

Research questions

• RQ1 What are the lexical, sentiment, and psycholinguistic differences between hate and counterspeech accounts on Twitter?
• RQ2 How do counterspeech strategies vary across target communities (e.g., religion, nationality, ethnicity, sexuality)?
• RQ3 Can account-level features reliably distinguish hate from counterspeech accounts, and which features contribute most to prediction?
• RQ4 What are the topical interests and personality traits that differentiate hateful and counterspeech accounts?

Key findings

• 1290 counterspeech replies were observed to 558 hate tweets, with counterspeech comprising 75.39% of replies.
• The best classifier (CatBoost) achieved 78% accuracy (F1 0.77) in distinguishing hate vs counterspeech accounts; XGBoost achieved 74% accuracy.
• Hate accounts tend to be older, more popular, and have more followers, while counterspeech accounts have more friends per day.
• Hate tweets from verified accounts show higher virality than non-verified hate tweets; verified hate accounts garnered substantially higher engagement metrics.
• Lexical analysis shows hate accounts use more envy, hate, negative emotion, profanity, while counterspeech accounts use more words related to government, law, and leadership; Empath and personality analyses show higher agreeableness in counterspeakers and higher extraversion in hate accounts.
• Topic analyses indicate counterspeech focuses more on politics, news, and journalism, while hate accounts concentrate on topics like racial slurs.