[Paper Review] ArCOV19-Rumors: Arabic COVID-19 Twitter Dataset for Misinformation Detection

Fatima Haouari, Maram Hasanain+2 more|arXiv (Cornell University)|Oct 17, 2020

Misinformation and Its Impacts39 citations

TL;DR

The authors introduce ArCOV19-Rumors, a manually annotated Arabic Twitter dataset for misinformation detection, supporting both claim-level and tweet-level verification with propagation networks, and provide baseline benchmarks using SOTA models.

ABSTRACT

In this paper we introduce ArCOV19-Rumors, an Arabic COVID-19 Twitter dataset for misinformation detection composed of tweets containing claims from 27th January till the end of April 2020. We collected 138 verified claims, mostly from popular fact-checking websites, and identified 9.4K relevant tweets to those claims. Tweets were manually-annotated by veracity to support research on misinformation detection, which is one of the major problems faced during a pandemic. ArCOV19-Rumors supports two levels of misinformation detection over Twitter: verifying free-text claims (called claim-level verification) and verifying claims expressed in tweets (called tweet-level verification). Our dataset covers, in addition to health, claims related to other topical categories that were influenced by COVID-19, namely, social, politics, sports, entertainment, and religious. Moreover, we present benchmarking results for tweet-level verification on the dataset. We experimented with SOTA models of versatile approaches that either exploit content, user profiles features, temporal features and propagation structure of the conversational threads for tweet verification.

Motivation & Objective

• Create a manually annotated Arabic Twitter dataset (ArCOV19-Rumors) for misinformation detection during COVID-19.
• Enable both claim-level verification (free-text claims) and tweet-level verification (claims within tweets).
• Provide propagation networks (retweets and conversation threads) to study spread and support verification.
• Cover diverse claim topics beyond health (social, political, religious, entertainment, sports).
• Offer baseline models and benchmarks to guide future research on Arabic misinformation detection.

Proposed method

• Collect verified COVID-19 claims (113 false, 49 true initially) from Arabic fact-checkers and health organizations, then balance with additional true claims from WHO/UN sources and English platforms (total 162 claims: 113 false, 49 true).
• For each claim, construct Boolean queries to retrieve relevant tweets from ArCOV-19 and manually filter for relevance (yielding 9,414 relevant tweets).
• Annotate each relevant tweet with stance relative to the claim: Expressing same claim, Negating the claim, or Other.
• Collect propagation networks (retweets and conversational threads) for each relevant tweet to enable analysis of spread and multi-modal verification signals.
• Define three tasks enabled by the dataset: claim-level verification, tweet-level verification (including early verification using prior relevant tweets), and claim retrieval of tweets expressing the same claim.
• Release dataset components including verified claims, claim-relevant tweets with labels, propagation networks, annotation guidelines, and baselines folds.

Experimental results

Research questions

• RQ1 How well can Arabic misinformation be detected at the claim level and at the tweet level using textual and contextual signals (including propagation networks)?
• RQ2 Can tweet-level verification leverage propagation networks, temporal features, and user profiles to improve veracity judgments in Arabic?
• RQ3 What is the distribution of true/false/other labels across claims and tweets, and how do propagation structures aid verification?
• RQ4 Does providing early-retrieval of relevant tweets (without full propagation of the target tweet) improve early verification performance?

Key findings

• The dataset includes 138 verified COVID-19 claims with 9,414 labeled relevant tweets across health and other topics (social, political, religious, entertainment, sports).
• Two levels of verification are supported: claim-level (verify the claim using all relevant tweets) and tweet-level (verify individual tweets).
• Benchmark experiments show that pre-trained Arabic BERT models (MARBERT, AraBERT) outperform other baselines in tweet-level verification, with MARBERT achieving macro-F1 around 0.74.
• Models leveraging propagation networks (Bi-GCN, PPC-RNN+CNN) outperform content-only approaches, but performance varies between true/false detection.
• The dataset includes propagation structures such as replies and retweets, enabling multi-modal and temporal analysis for misinformation detection in Arabic Twitter content.