[Paper Review] Characterizing and Predicting Social Correction on Twitter

Yingchen Ma, Bing He+2 more|arXiv (Cornell University)|Mar 15, 2023

Misinformation and Its Impacts16 citations

TL;DR

This paper builds a large dataset of COVID-19 vaccine misinformation and counter-misinformation replies on Twitter, analyzes linguistic/engagement/poster factors linked to social correction, and develops predictive models for if and how much countering will occur.

ABSTRACT

Online misinformation has been a serious threat to public health and society. Social media users are known to reply to misinformation posts with counter-misinformation messages, which have been shown to be effective in curbing the spread of misinformation. This is called social correction. However, the characteristics of tweets that attract social correction versus those that do not remain unknown. To close the gap, we focus on answering the following two research questions: (1) ``Given a tweet, will it be countered by other users?'', and (2) ``If yes, what will be the magnitude of countering it?''. This exploration will help develop mechanisms to guide users' misinformation correction efforts and to measure disparity across users who get corrected. In this work, we first create a novel dataset with 690,047 pairs of misinformation tweets and counter-misinformation replies. Then, stratified analysis of tweet linguistic and engagement features as well as tweet posters' user attributes are conducted to illustrate the factors that are significant in determining whether a tweet will get countered. Finally, predictive classifiers are created to predict the likelihood of a misinformation tweet to get countered and the degree to which that tweet will be countered. The code and data is accessible on https://github.com/claws-lab/social-correction-twitter.

Motivation & Objective

• Motivate understanding of which misinformation tweets attract social correction and why.
• Create a large paired dataset of misinformation tweets and counter-misinformation replies for COVID-19 vaccines.
• Perform stratified analysis of linguistic, engagement, and poster attributes to identify factors tied to countering.
• Develop predictive classifiers to forecast whether a tweet will be countered and the degree of countering.
• Provide insights into potential inequalities in social correction across user attributes.

Proposed method

• Curate a large-scale dataset of misinformation tweets and counter-misinformation replies (misinformation: 1,523,849 tweets; counter-replies: 690,047).
• Annotate a subset of counter-replies to train a counter-reply classifier using Roberta-base (lowercased).
• Train a misinformation tweet classifier (BERT) to identify misinformation within a large COVID-19 vaccine tweet corpus.
• Construct a stratified analysis by grouping tweets into strata based on their number of replies to compare highly vs. low countering within each stratum.
• Perform statistical and feature analyses across linguistic, engagement, and poster attributes (LIWC, VADER, politeness/impoliteness, engagement ratios, follower/verification metrics, and pre-misinformation activity).
• Build two prediction models to (RQ1) predict if a misinformation tweet will be countered, and (RQ2) predict the degree of countering (low vs high) using selected features.

Experimental results

Research questions

• RQ1 RQ1: Given a misinformation tweet, will it be countered by other users?
• RQ2 RQ2: If countered, what will be the magnitude of countering (low or high)?

Key findings

• A novel dataset with 1,523,849 misinformation tweets and 690,047 counter-misinformation replies was created, with a hand-annotated counter-reply subset.
• Within strata by reply count, highly countered tweets show stronger emotional content, including higher affect, negative sentiment, and anger, compared to low countered ones.
• Highly countered tweets exhibit more impoliteness and fewer health-related terms, suggesting topical and rhetoric factors influence correction.
• Engagement analysis reveals highly countered tweets receive fewer quotes, retweets, and likes per reply, indicating countering can curb broader spread.
• Poster attributes show verified posters have a higher likelihood of highly countered posts, but most attributes do not differ significantly for non-verified users.
• Education level of users (estimated via ARI on pre-misinformation posts) negatively correlates with countering likelihood, indicating lower-education-promoted accounts are more often countered.