[Paper Review] CoAID: COVID-19 Healthcare Misinformation Dataset

Limeng Cui, Dongwon Lee|arXiv (Cornell University)|May 22, 2020

Misinformation and Its ImpactsSocial Sciences21 references182 citations

TL;DR

Introduces the CoAID benchmark dataset for COVID-19 healthcare misinformation, including news articles, user engagement, and ground-truth labels across multiple platforms.

ABSTRACT

As the COVID-19 virus quickly spreads around the world, unfortunately, misinformation related to COVID-19 also gets created and spreads like wild fire. Such misinformation has caused confusion among people, disruptions in society, and even deadly consequences in health problems. To be able to understand, detect, and mitigate such COVID-19 misinformation, therefore, has not only deep intellectual values but also huge societal impacts. To help researchers combat COVID-19 health misinformation, therefore, we present CoAID (Covid-19 heAlthcare mIsinformation Dataset), with diverse COVID-19 healthcare misinformation, including fake news on websites and social platforms, along with users' social engagement about such news. CoAID includes 4,251 news, 296,000 related user engagements, 926 social platform posts about COVID-19, and ground truth labels. The dataset is available at: https://github.com/cuilimeng/CoAID.

Motivation & Objective

• Motivate the need to study COVID-19 misinformation and its societal impact.
• Provide a comprehensive, multi-modal dataset covering news articles, social posts, and user engagement.
• Enable benchmarking of misinformation detection models on real-world, healthcare-focused COVID-19 content.
• Demonstrate dataset construction, analysis, and baseline detection performance to guide future research.

Proposed method

• Dataset construction from reliable sources and fact-checkers to collect fake and true COVID-19 healthcare news.
• Crawling and aligning multi-modal data: news articles, short claims, social platform posts, and user engagement.
• Automatic updates to fetch latest information and ground-truth labels.
• Extraction of rich features per item including article content, metadata, and engagement signals like tweets and replies.
• Evaluation of multiple baseline and state-of-the-art misinformation detection models on the dataset.

Experimental results

Research questions

• RQ1 What are the distinguishing features between COVID-19 misinformation and factual information across websites and social platforms?
• RQ2 How do user engagement signals (tweets, replies, posts) contribute to misinformation detection accuracy on COVID-19 content?
• RQ3 What is the performance of various baseline and advanced models for COVID-19 healthcare misinformation detection using the CoAID dataset?

Key findings

• CoAID combines fake/true news, short claims, and extensive user engagement across five social platforms.
• State-of-the-art models leveraging article content and user engagement outperform simple baselines but face class imbalance and limited recall/F1.
• Misinformation detection performance varies across models; deeper, multi-modal approaches (e.g., SAMEv, dEFEND) yield better PR-AUC scores than text-only baselines.
• Dataset versions show growth over time, enabling analysis of misinformation trends and model robustness as data evolves.
• Public release and automatic updates offer a scalable benchmark for ongoing research into COVID-19 misinformation detection.