[Paper Review] Countering Misinformation on Social Media Through Educational Interventions: Evidence from a Randomized Experiment in Pakistan

Sajid Ali, Ihsan Ayyub Qazi|arXiv (Cornell University)|Jul 6, 2021

Misinformation and Its Impacts11 citations

TL;DR

This randomized experiment in urban Pakistan tests educational interventions to improve fake news detection among low-digital-literacy users. While general video education had no effect, personalized feedback after baseline assessments improved fake news identification by 0.14 standard deviations, highlighting the importance of tailored design in digital literacy programs.

ABSTRACT

Fake news is a growing problem in developing countries with potentially far-reaching consequences. We conduct a randomized experiment in urban Pakistan to evaluate the effectiveness of two educational interventions to counter misinformation among low-digital literacy populations. We do not find a significant effect of video-based general educational messages about misinformation. However, when such messages are augmented with personalized feedback based on individuals' past engagement with fake news, we find an improvement of 0.14 standard deviations in identifying fake news. We also find negative but insignificant effects on identifying true news, driven by female respondents. Our results suggest that educational interventions can enable information discernment but their effectiveness critically depends on how well their features and delivery are customized for the population of interest.

Motivation & Objective

• To assess the effectiveness of educational interventions in improving fake news detection among low-digital-literacy users in developing countries.
• To examine whether general educational videos alone can reduce misinformation susceptibility.
• To test whether personalized feedback based on past engagement with fake news enhances information discernment.
• To explore gender and literacy-based disparities in response to educational interventions.
• To evaluate the long-term persistence of intervention effects on belief formation and news evaluation.

Proposed method

• Conducted a clustered randomized control trial with 750 participants across 100 urban clusters in Lahore, Pakistan.
• Measured baseline fake news identification using real social media stories circulated in Pakistan.
• Delivered a general educational video in the national language to one treatment group to teach common features of misinformation.
• Provided personalized feedback to a second treatment group, linking their past responses to fake news with specific indicators of deception.
• Administered two endline surveys to assess changes in beliefs about new true and fake news stories.
• Used robustness checks to rule out experimenter demand effects and ensure result validity.

Experimental results

Research questions

• RQ1 Does a general video-based educational intervention significantly improve individuals' ability to identify fake news?
• RQ2 Does adding personalized feedback about past engagement with fake news enhance the effectiveness of educational interventions?
• RQ3 Are there gender-specific effects, particularly regarding skepticism toward true news?
• RQ4 How durable are the effects of educational interventions over time?
• RQ5 To what extent do educational interventions reduce belief in misinformation, especially on sensitive topics like COVID-19?

Key findings

• General video-based education alone had no statistically significant effect on identifying fake news.
• The combination of video education and personalized feedback improved fake news identification by 0.14 standard deviations.
• Both fake and true news identification improved by 0.076 standard deviations in the personalized feedback group.
• Female respondents showed a significant negative effect on identifying true news, suggesting increased skepticism toward accurate content.
• The intervention effect persisted over time, indicating durable learning effects.
• The study found lower take-up among less digitally literate individuals and women, indicating design challenges for equity.