[Paper Review] Facebook's Architecture Undermines Vaccine Misinformation Removal Efforts

David Broniatowski, Jiayan Gu+3 more|arXiv (Cornell University)|Feb 4, 2022

Misinformation and Its Impacts4 citations

TL;DR

This study investigates Facebook's vaccine misinformation removal policies and finds that while they reduced anti-vaccine posts, they also triggered unintended consequences: pro-vaccine content was mistakenly removed, remaining anti-vaccine content became more misinformative and polarized, and engagement rebounded to pre-policy levels due to the platform’s architecture that prioritizes community formation. The results suggest that Facebook’s design inadvertently enables persistent and amplified spread of vaccine misinformation despite content moderation efforts.

ABSTRACT

Misinformation promotes distrust in science, undermines public health, and may drive civil unrest. Vaccine misinformation, in particular, has stalled efforts to overcome the COVID-19 pandemic, prompting social media platforms' attempts to reduce it. Some have questioned whether "soft" content moderation remedies -- e.g., flagging and downranking misinformation -- were successful, suggesting that the addition of "hard" content remedies -- e.g., deplatforming and content bans -- is necessary. We therefore examined whether Facebook's vaccine misinformation content removal policies were effective. Here, we show that Facebook's policies reduced the number of anti-vaccine posts but also caused several perverse effects: pro-vaccine content was also removed, engagement with remaining anti-vaccine content repeatedly recovered to pre-policy levels, and this content became more misinformative, more politically polarised, and more likely to be seen in users' newsfeeds. We explain these results as an unintended consequence of Facebook's design goal: promoting community formation. Members of communities dedicated to vaccine refusal appear to seek out misinformation from multiple sources. Community administrators make use of several channels afforded by the Facebook platform to disseminate misinformation. Our findings suggest the need to address how social media platform architecture enables community formation and mobilisation around misinformative topics when managing the spread of online content.

Motivation & Objective

• To evaluate the effectiveness of Facebook’s content moderation policies in reducing vaccine misinformation.
• To investigate unintended consequences of Facebook’s moderation strategies on content visibility and user engagement.
• To examine how platform architecture, particularly community formation, enables the persistence and spread of vaccine misinformation.
• To understand how community administrators exploit platform features to disseminate misinformation across multiple sources.

Proposed method

• The researchers analyzed Facebook’s content moderation policies and their implementation between 2020 and 2021.
• They conducted a longitudinal analysis of anti-vaccine and pro-vaccine content visibility and engagement before and after policy enforcement.
• They examined user behavior and content patterns within anti-vaccine communities, focusing on cross-source sharing and platform feature utilization.
• They assessed changes in content quality, polarization, and algorithmic visibility using engagement metrics and content classification.
• They evaluated the role of community administrators in coordinating misinformation dissemination across multiple channels.
• They used qualitative and quantitative data from public Facebook content and policy reports to identify systemic design flaws.

Experimental results

Research questions

• RQ1 To what extent did Facebook’s content moderation policies reduce the volume of anti-vaccine content on the platform?
• RQ2 Why did engagement with remaining anti-vaccine content recover to pre-policy levels despite enforcement?
• RQ3 How did the platform’s architecture, particularly its community formation mechanisms, contribute to the persistence of vaccine misinformation?
• RQ4 What role did community administrators play in circumventing content moderation through multi-source dissemination?
• RQ5 How did the quality and polarization of remaining anti-vaccine content change after moderation?

Key findings

• Facebook’s moderation policies reduced the number of anti-vaccine posts, but the reduction was short-lived as engagement levels rebounded to pre-policy levels.
• Pro-vaccine content was frequently misclassified and removed, indicating flaws in content moderation algorithms.
• Remaining anti-vaccine content became more misinformative, more politically polarized, and more likely to appear in users’ newsfeeds after moderation.
• Anti-vaccine communities actively sought misinformation from multiple sources, exploiting Facebook’s platform features to maintain visibility.
• Community administrators used multiple channels and cross-posting strategies to circumvent content removal, sustaining the reach of misinformation.
• The platform’s design goal of promoting community formation inadvertently enabled coordinated mobilization around misinformative topics, undermining moderation outcomes.