[Paper Review] Fighting the COVID-19 Infodemic: Modeling the Perspective of Journalists, Fact-Checkers, Social Media Platforms, Policy Makers, and the Society

Firoj Alam, Shaden Shaar+15 more|arXiv (Cornell University)|Apr 30, 2020

Misinformation and Its Impacts4 citations

TL;DR

This paper introduces a multilingual, multi-stakeholder dataset of 16,000 manually annotated COVID-19 tweets in Arabic, Bulgarian, Dutch, and English, designed to support fine-grained disinformation analysis from the perspectives of journalists, fact-checkers, social media platforms, policymakers, and society. It demonstrates strong performance using multilingual and multitask learning with pretrained Transformers, achieving up to 87.7% F1 on multiclass classification for claim veracity and 81.8% F1 on call-to-action detection in English using XLM-RoBERTa.

ABSTRACT

With the emergence of the COVID-19 pandemic, the political and the medical aspects of disinformation merged as the problem got elevated to a whole new level to become the first global infodemic. Fighting this infodemic has been declared one of the most important focus areas of the World Health Organization, with dangers ranging from promoting fake cures, rumors, and conspiracy theories to spreading xenophobia and panic. Addressing the issue requires solving a number of challenging problems such as identifying messages containing claims, determining their check-worthiness and factuality, and their potential to do harm as well as the nature of that harm, to mention just a few. To address this gap, we release a large dataset of 16K manually annotated tweets for fine-grained disinformation analysis that (i) focuses on COVID-19, (ii) combines the perspectives and the interests of journalists, fact-checkers, social media platforms, policy makers, and society, and (iii) covers Arabic, Bulgarian, Dutch, and English. Finally, we show strong evaluation results using pretrained Transformers, thus confirming the practical utility of the dataset in monolingual vs. multilingual, and single task vs. multitask settings.

Motivation & Objective

• To address the global 'infodemic' during the COVID-19 pandemic by creating a dataset that captures disinformation from multiple stakeholder perspectives.
• To develop a comprehensive annotation schema that integrates the needs of journalists, fact-checkers, social media platforms, policymakers, and the public.
• To enable fine-grained disinformation analysis by modeling check-worthiness, veracity, harm, and calls to action in multilingual social media content.
• To evaluate the utility of the dataset through strong performance benchmarks using multilingual and multitask learning with state-of-the-art Transformer models.

Proposed method

• Annotating 16,000 tweets across four languages (Arabic, Bulgarian, Dutch, English) using a seven-question schema covering claim veracity, check-worthiness, harm, and policy relevance.
• Designing annotation guidelines through stakeholder consultations with journalists, fact-checkers, and a Ministry of Public Health to ensure real-world relevance.
• Employing a multi-label, multi-task learning framework to jointly predict claim veracity, check-worthiness, harm potential, and call-to-action intent.
• Training and evaluating multiple models including BERT, mBERT, XLM-RoBERTa, FastText, and BERTje on monolingual and multilingual settings.
• Incorporating social context and propagandistic language cues to improve model performance on disinformation detection.
• Releasing the dataset and code publicly to support reproducibility and further research in multilingual disinformation detection.

Experimental results

Research questions

• RQ1 How can a multilingual, multi-stakeholder dataset improve the detection of disinformation in social media during a global health crisis?
• RQ2 What performance gains can be achieved through multitask and cross-lingual learning on fine-grained disinformation classification tasks?
• RQ3 How do social context and linguistic cues such as propaganda affect the detection of harmful or check-worthy claims in tweets?
• RQ4 To what extent do pretrained Transformer models generalize across languages and tasks in the context of COVID-19 disinformation?
• RQ5 Can a unified annotation schema effectively capture the diverse needs of journalists, fact-checkers, policymakers, and the public?

Key findings

• The dataset achieved strong performance in multilingual disinformation detection, with XLM-RoBERTa attaining 87.7% F1 on multiclass claim veracity classification in English.
• Multitask learning significantly improved performance, with XLM-RoBERTa achieving 81.8% F1 on call-to-action detection (Q7) in English, outperforming baseline models.
• Cross-lingual transfer learning showed notable gains, with mBERT achieving 84.4% F1 on Q7 (call to action) in Dutch, indicating strong zero-shot generalization.
• Modeling propagandistic language and social context led to measurable improvements in detecting harmful and check-worthy claims across all languages.
• The dataset demonstrated high inter-annotator agreement and practical utility, with BERT-based models achieving over 80% F1 on most tasks in English and Dutch.
• The release of the dataset and code on GitHub enables reproducibility and supports future research in multilingual disinformation detection and policy-relevant NLP.