[Paper Review] Retweet-BERT: Political Leaning Detection Using Language Features and Information Diffusion on Social Networks

Julie Jiang, Xiang Ren+1 more|arXiv (Cornell University)|Jul 18, 2022

Misinformation and Its Impacts7 citations

TL;DR

Retweet-BERT is a scalable, unsupervised model that predicts Twitter users' political leanings by combining BERT-based embeddings of profile descriptions with retweet network structures, achieving 96–97% macro-F1 on two recent datasets; it reveals asymmetric echo chamber effects, with stronger polarization among right-leaning users during the COVID-19 discourse.

ABSTRACT

Estimating the political leanings of social media users is a challenging and ever more pressing problem given the increase in social media consumption. We introduce Retweet-BERT, a simple and scalable model to estimate the political leanings of Twitter users. Retweet-BERT leverages the retweet network structure and the language used in users' profile descriptions. Our assumptions stem from patterns of networks and linguistics homophily among people who share similar ideologies. Retweet-BERT demonstrates competitive performance against other state-of-the-art baselines, achieving 96%-97% macro-F1 on two recent Twitter datasets (a COVID-19 dataset and a 2020 United States presidential elections dataset). We also perform manual validation to validate the performance of Retweet-BERT on users not in the training data. Finally, in a case study of COVID-19, we illustrate the presence of political echo chambers on Twitter and show that it exists primarily among right-leaning users. Our code is open-sourced and our data is publicly available.

Motivation & Objective

• To develop a scalable, annotation-free method for estimating political ideology on Twitter using linguistic and network features.
• To investigate how information diffusion patterns and linguistic homophily correlate with political polarization on social media.
• To analyze the presence and structure of political echo chambers during high-stakes events like the 2020 U.S. presidential election and the COVID-19 pandemic.
• To validate the model’s performance on unseen users through manual evaluation and case studies.
• To provide open-source code and public data for reproducibility and further research on online polarization.

Proposed method

• Retrain BERT on user profile descriptions to extract contextualized linguistic embeddings reflecting ideology, sentiment, and affiliations.
• Construct a retweet network where edges represent retweets, with weights ≥2 to ensure meaningful endorsement signals.
• Apply unsupervised pre-training on the combined profile and network data to learn joint user representations.
• Fine-tune the model on a small, heuristically labeled subset using hashtags and media bias ratings from AllSides.com.
• Use logistic regression with hyperparameter tuning to classify political polarity based on learned embeddings.
• Leverage node2vec and GraphSAGE as baseline models for comparison, with consistent training and evaluation protocols.

Experimental results

Research questions

• RQ1 Can linguistic homophily and retweet network structures jointly improve political leaning detection on Twitter?
• RQ2 How does Retweet-BERT perform compared to state-of-the-art models on recent, large-scale Twitter datasets?
• RQ3 To what extent do echo chambers form in political discourse, and are they asymmetric across left- and right-leaning users?
• RQ4 How effective is Retweet-BERT in generalizing to users not in the training data, as validated through manual evaluation?
• RQ5 What patterns of information diffusion emerge in politically polarized communities during high-impact events like the COVID-19 pandemic?

Key findings

• Retweet-BERT achieves 96% macro-F1 on the COVID-19 Twitter dataset and 97% macro-F1 on the 2020 U.S. presidential election dataset, outperforming state-of-the-art baselines.
• The model demonstrates strong generalization, with manual validation confirming reliable performance on users not included in the training data.
• Right-leaning users exhibit significantly stronger echo chamber effects, with more tightly clustered retweet networks and higher homophily.
• Left-leaning users show more diverse information exposure, indicating less structural isolation in their retweet networks.
• The presence of political echo chambers is most pronounced among right-leaning users, particularly during the COVID-19 discourse.
• Heuristically labeled hashtags and media bias ratings from AllSides.com effectively enable pseudo-labeling without human-annotated data.