[Paper Review] NewsQA: A Machine Comprehension Dataset

Adam Trischler, Tong Wang+5 more|arXiv (Cornell University)|Nov 29, 2016

Topic Modeling17 references77 citations

TL;DR

NewsQA is a large-scale, crowdsourced machine comprehension dataset comprising over 119,000 natural language questions and span-based answers derived from CNN news articles. It emphasizes reasoning through a four-stage collection process, resulting in a significant performance gap (0.198 F1) between humans and state-of-the-art neural models, highlighting its challenge for advancing machine comprehension systems.

ABSTRACT

We present NewsQA, a challenging machine comprehension dataset of over 100,000 human-generated question-answer pairs. Crowdworkers supply questions and answers based on a set of over 10,000 news articles from CNN, with answers consisting of spans of text from the corresponding articles. We collect this dataset through a four-stage process designed to solicit exploratory questions that require reasoning. A thorough analysis confirms that NewsQA demands abilities beyond simple word matching and recognizing textual entailment. We measure human performance on the dataset and compare it to several strong neural models. The performance gap between humans and machines (0.198 in F1) indicates that significant progress can be made on NewsQA through future research. The dataset is freely available at https://datasets.maluuba.com/NewsQA.

Motivation & Objective

• To create a large-scale, natural-language machine comprehension dataset that captures complex reasoning beyond simple word matching.
• To address the limitations of existing datasets—either too small for deep learning or synthetically generated—by using human-posed, exploratory questions.
• To develop a dataset that reflects real-world information-seeking behavior, requiring synthesis and inference across document spans.
• To provide a benchmark that reveals the current limitations of neural models in reasoning and comprehension tasks.
• To enable future research in building more literate artificial intelligence systems through a challenging, realistic dataset.

Proposed method

• Employed a four-stage crowd-sourcing process to elicit exploratory, curiosity-driven questions from crowdworkers based on CNN news articles.
• Collected questions and answers as span-level text excerpts from the original articles, ensuring answers are directly extractable from the text.
• Designed the collection process to encourage lexical and syntactic divergence between questions and answers, reducing reliance on surface-level matching.
• Introduced a null answer option for questions with no answer in the article, increasing realism and challenge.
• Used inverse sentence frequency (isf) as a baseline method to evaluate sentence-level answer retrieval, measuring performance on both NewsQA and artificially lengthened SQuAD articles.
• Evaluated models using standard metrics (EM, F1, BLEU, CIDEr) on a held-out dev set, comparing performance across answer types and reasoning categories.

Experimental results

Research questions

• RQ1 Can a large-scale, human-posed machine comprehension dataset with reasoning-intensive questions improve the evaluation of neural question-answering models?
• RQ2 To what extent do current neural models fail to generalize beyond surface-level matching when faced with complex, natural-language questions requiring inference?
• RQ3 How does the performance gap between humans and models on NewsQA compare to that on SQuAD, and what does this imply for future model development?
• RQ4 Does increasing document length significantly degrade performance on simple retrieval baselines like inverse sentence frequency (isf), and does this reflect the true difficulty of NewsQA?
• RQ5 How do different answer types (e.g., named entities vs. descriptive spans) and reasoning types (e.g., word matching vs. synthesis) affect model performance on NewsQA?

Key findings

• Human performance on NewsQA achieved an F1 score of 0.820, significantly outperforming the best neural model (BARB) with an F1 of 0.622, resulting in a 0.198 F1 gap.
• The performance gap on NewsQA (0.198 F1) is more than double that on SQuAD (0.098 F1), indicating that NewsQA poses a substantially greater challenge to current models.
• The baseline model BARB achieved only 35.4% sentence-level accuracy on NewsQA using the isf method, compared to 79.6% on SQuAD, even when SQuAD articles were artificially lengthened to match NewsQA’s average length.
• Model performance was lowest on questions requiring synthesis and inference, with F1 dropping significantly compared to word-matching tasks, suggesting difficulty in tracking long-range dependencies.
• Humans outperformed models on ambiguous and incomplete questions, a pattern not observed in SQuAD, indicating that NewsQA better captures real-world comprehension complexity.
• The dataset contains 119,633 questions across 12,744 articles, with 18.5% of questions having no answer in the article (null span), and answers span multiple words or phrases, increasing complexity.