[Paper Review] What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams

Di Jin, Eileen Pan+4 more|arXiv (Cornell University)|Sep 28, 2020

Topic ModelingComputer Science30 references60 citations

TL;DR

This paper introduces MedQA, a large-scale, multilingual open-domain multiple-choice QA dataset built from medical board exams, with a text corpus of medical textbooks and baseline experiments showing significant challenges for current OpenQA systems.

ABSTRACT

Open domain question answering (OpenQA) tasks have been recently attracting more and more attention from the natural language processing (NLP) community. In this work, we present the first free-form multiple-choice OpenQA dataset for solving medical problems, MedQA, collected from the professional medical board exams. It covers three languages: English, simplified Chinese, and traditional Chinese, and contains 12,723, 34,251, and 14,123 questions for the three languages, respectively. We implement both rule-based and popular neural methods by sequentially combining a document retriever and a machine comprehension model. Through experiments, we find that even the current best method can only achieve 36.7\%, 42.0\%, and 70.1\% of test accuracy on the English, traditional Chinese, and simplified Chinese questions, respectively. We expect MedQA to present great challenges to existing OpenQA systems and hope that it can serve as a platform to promote much stronger OpenQA models from the NLP community in the future.

Motivation & Objective

• Create a free-form multiple-choice OpenQA dataset for medical problems sourced from US, Mainland China, and Taiwan medical board exams.
• Provide a large medical textbook corpus to support reading comprehension over retrieved evidence.
• Evaluate state-of-the-art OpenQA approaches to identify bottlenecks and guide future model development in retrieval and reasoning.
• Analyze dataset characteristics to highlight challenges such as multi-hop reasoning and domain-specific knowledge requirements.

Proposed method

• Construct a two-component OpenQA system consisting of a document retriever and a document reader following the DrQA framework.
• Compare rule-based baselines (PMI, IR with BM25) against neural models including fine-tuned BERT/Roberta family variants.
• Use both English and Chinese (simplified/traditional) questions with four options per question; randomize option order and adjust data for fair use.
• Fine-tune pretrained language models (e.g., BERT, BioBERT, RoBERTa) on MedQA with input formed as [CLS] context [SEP] question+options [SEP].
• Use a large collection of English and Chinese medical textbooks as the knowledge source; perform OCR on PDFs and preprocessing before parsing.

Experimental results

Research questions

• RQ1 How well do current OpenQA systems perform on a large-scale, domain-specific medical OpenQA dataset across multiple languages?
• RQ2 What are the main bottlenecks in retrieval and reading comprehension when solving complex medical case questions?
• RQ3 How does multilingual MedQA affect model performance and cross-lingual transfer in OpenQA?
• RQ4 To what extent do retrieved passages provide sufficient evidence to answer multi-hop medical questions?
• RQ5 What insights do error analyses provide for improving retrieval and reasoning in medical OpenQA models?

Key findings

• Even the strongest models achieve limited accuracy on MedQA, with 36.7% (US), 42.0% (Traditional Chinese), and 70.1% (Simplified Chinese) on test sets for the best methods.
• Retrieval quality is the bottleneck, as current IR/retrieval fails to sustain multi-hop reasoning over evidence spanning multiple passages.
• Pretrained language models generally outperform non-pretrained baselines on MCMLE, but many fail to surpass IR baselines on USMLE and TWMLE, underscoring dataset difficulty.
• The dataset exhibits two question types (single knowledge point vs. multi-hop, case-based reasoning) with a higher prevalence of the challenging Type 2 questions in USMLE.
• Human experts can find sufficient evidence in the medical textbook corpus for a high percentage of questions (USMLE 88%, MCMLE 100%, TWMLE 87%), suggesting the corpus coverage is adequate for many items.
• Error analyses reveal retrieval failures as a major cause of incorrect answers on USMLE, with two failure patterns: overly broad candidate diseases and two-step reasoning requirements.