[Paper Review] End-to-End QA on COVID-19: Domain Adaptation with Synthetic Training

Revanth Gangi Reddy, Bhavani Iyer+6 more|arXiv (Cornell University)|Dec 2, 2020

Topic ModelingComputer Science43 references17 citations

TL;DR

This paper proposes a zero-shot domain adaptation framework for end-to-end question answering on the COVID-19 domain using synthetically generated training data. By leveraging a pre-trained question generator on open-domain QA data to create synthetic question-passage-answer triples from in-domain CORD-19 texts, the authors fine-tune both neural IR and MRC models, achieving significant F1 improvements over a state-of-the-art open-domain baseline on multiple COVID-19 QA benchmarks.

ABSTRACT

End-to-end question answering (QA) requires both information retrieval (IR) over a large document collection and machine reading comprehension (MRC) on the retrieved passages. Recent work has successfully trained neural IR systems using only supervised question answering (QA) examples from open-domain datasets. However, despite impressive performance on Wikipedia, neural IR lags behind traditional term matching approaches such as BM25 in more specific and specialized target domains such as COVID-19. Furthermore, given little or no labeled data, effective adaptation of QA systems can also be challenging in such target domains. In this work, we explore the application of synthetically generated QA examples to improve performance on closed-domain retrieval and MRC. We combine our neural IR and MRC systems and show significant improvements in end-to-end QA on the CORD-19 collection over a state-of-the-art open-domain QA baseline.

Motivation & Objective

• To address the challenge of poor performance of neural IR and MRC models in specialized domains like COVID-19, where labeled data is scarce.
• To develop a method for generating high-quality synthetic training examples for both information retrieval and machine reading comprehension in low-resource target domains.
• To evaluate the effectiveness of synthetic data in improving end-to-end QA performance on the CORD-19 corpus compared to open-domain baselines.
• To investigate whether synthetic examples trained on diverse question styles (e.g., SQuAD vs. NQ) yield better domain adaptation results.

Proposed method

• A synthetic question generator is trained on open-domain MRC datasets (e.g., SQuAD) to produce question-passage-answer triples from in-domain CORD-19 scientific articles.
• The generator uses top-p and top-k sampling techniques to improve diversity and quality of generated questions.
• Synthetic examples are used to fine-tune a pre-trained dense passage retriever (DPR) for improved retrieval in the target domain.
• A separate MRC model is fine-tuned using both in-domain language modeling on CORD-19 and synthetic MRC examples to improve answer extraction.
• A roundtrip consistency check is applied to filter low-quality synthetic examples and improve label reliability.
• End-to-end QA performance is evaluated by combining the adapted IR and MRC models using a weighted fusion of IR and MRC scores (IR weight = 0.7).

Experimental results

Research questions

• RQ1 Can synthetic question-answer pairs generated from in-domain text improve neural IR performance in low-resource domains like COVID-19?
• RQ2 Does fine-tuning the MRC model with synthetic examples lead to measurable improvements in answer extraction accuracy on domain-specific QA datasets?
• RQ3 How does the quality of synthetic examples differ when generated from question styles like SQuAD versus Natural Questions, and which yields better downstream performance?
• RQ4 To what extent do language modeling on the target domain and synthetic data generation jointly contribute to MRC improvement?
• RQ5 Is the proposed end-to-end QA system significantly better than a strong open-domain baseline in zero-shot adaptation to the COVID-19 domain?

Key findings

• The end-to-end QA system using synthetic data achieved a 47.8 F1 score on the test set of COVID-QA-111, a 1.9-point improvement over the baseline (45.9 F1).
• On the COVID-QA-2019 dataset, the final system achieved a 44.9 F1 score, representing a 3.7-point gain over the baseline (41.2 F1).
• The adapted MRC model improved by 3.1 EM and 2.6 F1 points on the dev set of COVID-QA-2019 when combining synthetic examples and in-domain language modeling.
• Using synthetic examples from a SQuAD-trained generator outperformed NQ-style generation, indicating that question style matters for synthetic data quality.
• A paired t-test confirmed that the final end-to-end system's performance was significantly better than the baseline at p < 0.01.
• The contribution of synthetic MRC examples (3.1 EM, 2.6 F1) exceeded that of in-domain language modeling (1.5 EM, 0.8 F1), highlighting their greater impact.