[Paper Review] SQuAD: 100,000+ Questions for Machine Comprehension of Text

Pranav Rajpurkar, Jian Zhang+2 more|arXiv (Cornell University)|Jun 16, 2016

Topic Modeling27 references822 citations

TL;DR

SQuAD introduces a large-scale reading comprehension dataset with crowd-sourced question-answer pairs where answers are spans from Wikipedia passages; a logistic regression model achieves 51.0% F1, well below human 86.8% F1, highlighting substantial room for improvement.

ABSTRACT

We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com

Motivation & Objective

• Provide a large, high-quality dataset for reading comprehension to advance machine understanding of text.
• Analyze the types of reasoning required by questions and the syntactic challenges involved.
• Establish a baseline model and evaluate human vs. machine performance on span-based answers.

Proposed method

• Crowd-source 107,785 question-answer pairs on 536 Wikipedia paragraphs where answers are text spans.
• Develop a logistic regression model with lexicalized and dependency-tree path features to predict answer spans.
• Constrain candidate answers to constituents via a constituency parse to enable efficient scoring.
• Evaluate using exact match and F1 metrics, with ablations to identify key features.
• Compare performance against sliding-window baselines and report human performance for reference.

Experimental results

Research questions

• RQ1 What kinds of reasoning and syntactic divergences are involved in SQuAD questions?
• RQ2 How well can a baseline machine learning model identify answer spans compared to humans?
• RQ3 Which features drive effective span-based question answering in this dataset?

Key findings

• SQuAD contains 107,785 Q&A pairs on 536 articles, with answers as span text from passages.
• The best logistic regression model achieves 51.0% F1 on v1.0 development/test, far from human 86.8% F1.
• Lexicalized features and dependency-path features are the most important for performance.
• Performance degrades with greater syntactic divergence between question and answer sentence, unlike humans.
• About 79.3% of correct answers are within the sentence containing the answer, indicating the main challenge is locating the exact span.
• SQuAD is substantially larger than prior manually labeled RC datasets and provides a robust challenge for future models.