[Paper Review] Rapidly Bootstrapping a Question Answering Dataset for COVID-19

Raphael Tang, Rodrigo Nogueira+5 more|arXiv (Cornell University)|Apr 23, 2020

Topic ModelingComputer Science17 references62 citations

TL;DR

CovidQA is a small, hand-crafted question–answering dataset for COVID-19 built from Kaggle’s CORD-19 reviews, designed to evaluate zero-shot and transfer learning of QA models on COVID-19 topics. Baselines show BM25 is strong, with domain-adapted transformers offering benefits in some settings.

ABSTRACT

We present CovidQA, the beginnings of a question answering dataset specifically designed for COVID-19, built by hand from knowledge gathered from Kaggle's COVID-19 Open Research Dataset Challenge. To our knowledge, this is the first publicly available resource of its type, and intended as a stopgap measure for guiding research until more substantial evaluation resources become available. While this dataset, comprising 124 question-article pairs as of the present version 0.1 release, does not have sufficient examples for supervised machine learning, we believe that it can be helpful for evaluating the zero-shot or transfer capabilities of existing models on topics specifically related to COVID-19. This paper describes our methodology for constructing the dataset and presents the effectiveness of a number of baselines, including term-based techniques and various transformer-based models. The dataset is available at http://covidqa.ai/

Motivation & Objective

• Provide a stopgap COVID-19 in-domain QA test set derived from Kaggle’s COVID-19 Open Research Dataset Challenge (CORD-19).
• Evaluate unsupervised and out-of-domain supervised QA models on COVID-19 content.
• Estimate how well domain adaptation and question form (natural language vs. keyword) affect QA performance.

Proposed method

• Construct 124 question–article pairs (version 0.1) from Kaggle’s literature reviews in CORD-19.
• For each pair, manually identify the exact answer span within the corresponding article; when exact spans are ambiguous, mark sentence-level relevance instead.
• Represent articles as sentences and score relevance to a query q via a scoring function ρ(q, s_i).
• Evaluate ranking using mean reciprocal rank (MRR), precision at 1 (P@1), and recall at 3 (R@3).
• Compare baselines including BM25, vanilla and domain-adapted BERT variants (SciBERT, BioBERT), and out-of-domain supervised models (SQuAD-finetuned BioBERT, MS MARCO-finetuned BERT/BioBERT, T5).
• Adopt a two-stage information access viewpoint (retrieval followed by highlight/QA pass) to mirror practical systems.

Experimental results

Research questions

• RQ1 How effective are zero-shot or transfer-based QA approaches at locating COVID-19 relevant answer sentences in scientific articles?
• RQ2 What is the added value of domain-adapted transformers versus traditional term-based baselines on COVID-19 content?
• RQ3 How do natural language questions compare to keyword queries in guiding QA and retrieval models in this domain?
• RQ4 What practical lessons arise for rapidly constructing evaluation resources during a pandemic?

Key findings

• BM25 is a strong baseline, outperforming several unsupervised neural approaches in both natural language and keyword-query settings.
• BioBERT (unsupervised) improves over vanilla BERT, especially for keyword queries, indicating domain adaptation helps with some input styles.
• Out-of-domain supervised models (fine-tuned on MS MARCO or SQuAD) outperform unsupervised counterparts, with T5 generally the most effective among tested models.
• Domain adaptation benefits are nuanced: BioBERT helps with NL questions, but MS MARCO fine-tuning can negate some BioBERT advantages.
• Well-formed natural language questions tend to yield better results for transformer models than keyword queries, highlighting design considerations for multi-stage retrieval pipelines.