[Paper Review] Birdwatch: Crowd Wisdom and Bridging Algorithms can Inform Understanding and Reduce the Spread of Misinformation

Stefan Wojcik, Sophie Hilgard+6 more|arXiv (Cornell University)|Oct 27, 2022

Misinformation and Its Impacts34 citations

TL;DR

The paper develops a matrix-factorization–based bridging algorithm to select crowd-sourced annotations that are informative across diverse viewpoints, then shows via survey and real deployment that these annotations improve understanding and reduce sharing of potentially misleading posts on Twitter.

ABSTRACT

We present an approach for selecting objectively informative and subjectively helpful annotations to social media posts. We draw on data from on an online environment where contributors annotate misinformation and simultaneously rate the contributions of others. Our algorithm uses a matrix-factorization (MF) based approach to identify annotations that appeal broadly across heterogeneous user groups - sometimes referred to as "bridging-based ranking." We pair these data with a survey experiment in which individuals are randomly assigned to see annotations to posts. We find that annotations selected by the algorithm improve key indicators compared with overall average and crowd-generated baselines. Further, when deployed on Twitter, people who saw annotations selected through this bridging-based approach were significantly less likely to reshare social media posts than those who did not see the annotations.

Motivation & Objective

• Motivate the need for high-quality, broadly informative crowd annotations to combat misinformation on social media.
• Develop an MF-based bridging algorithm to identify notes that appeal across diverse user viewpoints.
• Evaluate whether algorithm-selected notes enhance informativeness and perceived helpfulness.
• Assess the impact of annotated posts on users' sharing behaviors in a real-world Twitter deployment.

Proposed method

• Construct a sparse rater-note matrix from Birdwatch data with entries r_un in {0,1,null}.
• Factorize the matrix to learn user and note latent vectors f_u and f_n, plus intercepts μ, i_u, i_n.
• Use a regularized least-squares objective to learn parameters with strong regularization on intercepts to emphasize breadth of appeal.
• Label notes as 'helpful', 'not helpful', or 'needs more ratings' based on note intercept i_n thresholds (≥0.40, ≤-0.08, otherwise).
• Implement a rater helpfulness filter to remove low-quality raters before re-running MF and final labeling.
• Evaluate using three survey waves (randomized exposure to notes) to measure informativeness and helpfulness, and compare with baselines (average notes, supermajority).

Experimental results

Research questions

• RQ1 RQ1: Can algorithm-selected Birdwatch notes inform understanding and be broadly helpful across diverse political viewpoints better than a supermajority baseline?
• RQ2 RQ2: Are appraisals of a Tweet's accuracy a reliable proxy for agreement with a Tweet's claims?
• RQ3 RQ3: Do crowd-generated annotations reduce the spread of potentially misleading information on Twitter?

Key findings

• Notes selected by the bridging-based MF algorithm are perceived as helpful across diverse political viewpoints.
• Algorithmically selected notes improve informativeness about potentially misleading Tweets compared with overall averages and crowd baselines.
• Exposure to algorithm-selected annotations leads to users being less likely to reshare or Like the annotated posts.
• The survey-based evidence aligns with causal assessments of readers’ understanding when exposed to informative annotations.
• A bridging-based scoring approach can identify crowd annotations that are broadly informative and reduce sharing behaviors in a real deployment.