[Paper Review] Reranking partisan animosity in algorithmic social media feeds alters affective polarization

Tiziano Piccardi, Martin Saveski+4 more|arXiv (Cornell University)|Nov 22, 2024

Social Media and Politics4 citations

TL;DR

This study uses real-time LLM-powered feed reranking to test how exposure to antidemocratic attitudes and partisan animosity (AAPA) affects affective polarization on X/Twitter. In a 10-day preregistered field experiment with 1,256 users, reducing AAPA exposure significantly improved outgroup feelings and reduced negative emotions, while engagement metrics remained unchanged, suggesting algorithmic design can mitigate polarization without altering user behavior.

ABSTRACT

Today, social media platforms hold sole power to study the effects of feed ranking algorithms. We developed a platform-independent method that reranks participants' feeds in real-time and used this method to conduct a preregistered 10-day field experiment with 1,256 participants on X during the 2024 U.S. presidential campaign. Our experiment used a large language model to rerank posts that expressed antidemocratic attitudes and partisan animosity (AAPA). Decreasing or increasing AAPA exposure shifted out-party partisan animosity by two points on a 100-point feeling thermometer, with no detectable differences across party lines, providing causal evidence that exposure to AAPA content alters affective polarization. This work establishes a method to study feed algorithms without requiring platform cooperation, enabling independent evaluation of ranking interventions in naturalistic settings.

Motivation & Objective

• To investigate whether algorithmic exposure to antidemocratic attitudes and partisan animosity (AAPA) drives affective polarization on social media.
• To test whether real-time, content-based feed reranking using LLMs can reduce affective polarization in a field setting.
• To assess the impact of AAPA exposure on emotional states and traditional engagement metrics like reposts and favorites.
• To evaluate whether users perceive changes in their social media experience due to the intervention.
• To explore whether the intervention effects vary across demographic or socioeconomic subgroups.

Proposed method

• Developed an LLM-based classifier to detect eight categories of AAPA—antidemocratic attitudes and partisan animosity—in real-time social media content.
• Implemented a web extension that reranked participants’ X/Twitter feeds to either increase or decrease exposure to AAPA content based on the classifier’s output.
• Conducted a 10-day, preregistered field experiment with 1,256 consenting participants on X/Twitter, randomly assigning them to increased or reduced AAPA exposure conditions.
• Measured affective polarization using in-feed and post-experiment surveys, capturing feelings toward the political outgroup.
• Collected and analyzed engagement metrics (reposts, favorites, new posts/replies) to assess behavioral impacts.
• Used BERT-based embeddings and logistic regression to test whether users could detect changes in their feed experience, with cross-validation for model evaluation.

Experimental results

Research questions

• RQ1 Does reducing exposure to AAPA in algorithmic social media feeds lead to more positive feelings toward the political outgroup?
• RQ2 Does increasing exposure to AAPA content result in more negative emotional states, such as anger and sadness?
• RQ3 Do changes in AAPA exposure significantly affect traditional engagement metrics like reposts and favorites on X/Twitter?
• RQ4 Are there detectable differences in user perception of their Twitter experience between treatment and control groups?
• RQ5 Do the effects of AAPA exposure vary across demographic or socioeconomic subgroups?

Key findings

• Reducing exposure to AAPA in social media feeds led to significantly warmer feelings toward the political outgroup, with a treatment effect of β = 0.18 (95% CI [0.08, 0.28]) on in-feed affective polarization measures.
• Increasing exposure to AAPA content resulted in significantly colder feelings toward the outgroup, with a treatment effect of β = -0.16 (95% CI [-0.26, -0.06]) on in-feed affective polarization.
• Participants exposed to higher levels of AAPA reported a measurable increase in negative emotions, including anger and sadness, as captured in in-feed surveys.
• There was no statistically significant change in engagement metrics: the favorite rate changed by only 0.14% in the increased exposure group (p = 0.796) and decreased by 0.29% in the reduced exposure group (p = 0.43).
• The number of new posts and replies showed no significant differences between treatment and control groups.
• Participants were largely unable to detect the intervention: only 10.3% reported a positive experience, 4.1% noticed more political content (evenly distributed), and 2.8% reported performance slowdowns; classification models achieved only 50.5% F1 score in predicting experimental condition, indicating no strong perceptual signal.