[Paper Review] Extracting a Knowledge Base of COVID-19 Events from Social Media

Shi Zong, Ashutosh Baheti+2 more|arXiv (Cornell University)|Jun 3, 2020

Misinformation and Its Impacts32 references4 citations

TL;DR

This paper presents a manually annotated corpus of 10,000 COVID-19-related tweets with 28 fine-grained slots across five event types—positive/negative tests, deaths, denied testing access, and claimed cures/preventions. Using BERT-based classifiers fine-tuned on the corpus, the authors extract structured events from millions of tweets to build CovidKB, a knowledge base that supports high-precision, complex queries like 'Which organizations in Houston have employees who tested positive?'

ABSTRACT

In this paper, we present a manually annotated corpus of 10,000 tweets containing public reports of five COVID-19 events, including positive and negative tests, deaths, denied access to testing, claimed cures and preventions. We designed slot-filling questions for each event type and annotated a total of 31 fine-grained slots, such as the location of events, recent travel, and close contacts. We show that our corpus can support fine-tuning BERT-based classifiers to automatically extract publicly reported events and help track the spread of a new disease. We also demonstrate that, by aggregating events extracted from millions of tweets, we achieve surprisingly high precision when answering complex queries, such as "Which organizations have employees that tested positive in Philadelphia?" We will release our corpus (with user-information removed), automatic extraction models, and the corresponding knowledge base to the research community.

Motivation & Objective

• Address the challenge of extracting timely, granular public health information from social media during the COVID-19 pandemic.
• Overcome the limitations of official data, which often lags and lacks individual-level or organizational-level granularity.
• Develop a high-quality, linguistically rich dataset to support supervised event extraction for emerging crises.
• Construct a scalable knowledge base that enables complex, structured queries over publicly reported events from Twitter.
• Support public health monitoring, misinformation tracking, and decision-making for epidemiologists, journalists, and policymakers.

Proposed method

• Design five event types: positive test, negative test, denied testing access, death, and claimed cure/prevention.
• Create slot-filling questions for 28 fine-grained semantic slots (e.g., location, travel history, close contacts) to guide human annotation.
• Collect and annotate 10,000 tweets with span-level labeling for each slot, ensuring linguistic precision and consistency.
• Fine-tune BERT-based sequence classification models on the annotated corpus to automatically extract structured event information.
• Aggregate extractions across millions of tweets to improve precision by leveraging redundancy in widely discussed events.
• Index extracted events into a searchable knowledge base (CovidKB) that supports complex, structured queries using natural language-like syntax.

Experimental results

Research questions

• RQ1 Can a manually annotated, fine-grained corpus of social media tweets support effective training of BERT-based models for structured event extraction?
• RQ2 To what extent can aggregated extractions from large-scale Twitter data achieve high precision in identifying real-world public health events?
• RQ3 Can the resulting knowledge base answer complex, multi-fact structured queries with high accuracy, such as those involving location, organization, and exposure history?
• RQ4 How effective is the system in detecting and indexing claims about cures and preventions, which may help track misinformation?
• RQ5 To what extent can this methodology be rapidly adapted to other emerging crises, such as natural disasters or future disease outbreaks?

Key findings

• The corpus achieves slot F1 scores ranging from 0.3 to 0.9, with most exceeding 0.5, demonstrating the feasibility of fine-grained annotation for event extraction.
• Aggregating extractions across millions of tweets enables high-precision answers to complex queries, such as identifying organizations with employees who tested positive in Philadelphia.
• The knowledge base, CovidKB, indexes over 4.2 million events extracted from two years of Twitter data, supporting real-time, structured querying.
• The system successfully detects and tracks evolving public discourse on treatments, including sustained mentions of hydroxychloroquine and masks, and rising attention to vaccines and drugs like fluvoxamine and monoclonal antibodies.
• The system identifies prominent promoters of cures, including public figures (e.g., Donald Trump, Bill Gates), organizations (e.g., CDC, Pfizer), and controversial figures (e.g., Jim Bakker), highlighting potential misinformation vectors.
• Despite limitations such as sampling bias (1% of public tweets) and potential performance gaps on non-standard dialects, the system demonstrates strong utility for public health monitoring and crisis response.