[Paper Review] A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

Danqi Chen, Jason Bolton+1 more|arXiv (Cornell University)|Jun 9, 2016

Topic Modeling16 references128 citations

TL;DR

The paper analyzes the CNN/Daily Mail RC task and shows that simple, well-designed systems achieve state-of-the-art results (up to 76.6% on Daily Mail) and argues that the task is easier than previously thought with most questions solvable by single-sentence reasoning.

ABSTRACT

Enabling a computer to understand a document so that it can answer comprehension questions is a central, yet unsolved goal of NLP. A key factor impeding its solution by machine learned systems is the limited availability of human-annotated data. Hermann et al. (2015) seek to solve this problem by creating over a million training examples by pairing CNN and Daily Mail news articles with their summarized bullet points, and show that a neural network can then be trained to give good performance on this task. In this paper, we conduct a thorough examination of this new reading comprehension task. Our primary aim is to understand what depth of language understanding is required to do well on this task. We approach this from one side by doing a careful hand-analysis of a small subset of the problems and from the other by showing that simple, carefully designed systems can obtain accuracies of 73.6% and 76.6% on these two datasets, exceeding current state-of-the-art results by 7-10% and approaching what we believe is the ceiling for performance on this task.

Motivation & Objective

• Assess the difficulty of the CNN/Daily Mail RC task created from CNN/Daily Mail articles and bullet-point summaries.
• Identify which language understanding capabilities are required to perform well on these datasets.
• Develop and evaluate simple feature-based and neural models to establish lower-bounds and upper-bounds on performance.
• Diagnose data quality issues (coreference errors, anonymization) and their impact on model performance.

Proposed method

• Implement an entity-centric feature-based classifier and an end-to-end neural network inspired by AttentiveReader.
• Use a bilinear attention mechanism to compute relevance between question and passage contextual embeddings.
• Train with a softmax over candidate entities and optimize negative log-likelihood.
• Relabel entity markers by first occurrence to improve training efficiency and performance.
• Compare with window-based memory networks and prior RC models to assess task difficulty and ceiling performance.

Experimental results

Research questions

• RQ1 What level of natural language understanding is required to perform well on the CNN/Daily Mail RC task?
• RQ2 Are simple conventional NLP features competitive with neural models on this dataset?
• RQ3 What is the upper bound on performance given the dataset construction and coreference/anonymization issues?
• RQ4 How do model predictions break down by question type and linguistic phenomena (paraphrase, exact-match, coreference errors)?

Key findings

• A conventional feature-based classifier attains 67.9% accuracy on CNN test, surpassing previous symbolic baselines and many neural baselines.
• A neural AttentiveReader-like model achieves 72.5% (CNN) and 76.9% (Daily Mail) without relabeling; relabeling boosts to 73.6% (CNN) and 76.6% (Daily Mail).
• Ensembling five models yields additional gains, reaching up to 79.2% on Daily Mail and 80.2% on CNN for relabeled ensembles.
• Feature ablation shows n-gram match and entity frequency are the most impactful features in the classifier.
• Per-category analysis indicates exact-match questions are easy for both systems, whereas paraphrase and partial-clue questions show larger gains for neural models; coreference errors and hard/ambiguous cases limit ceiling performance to around 75%–80%.
• The authors argue the task largely reduces to single-sentence reasoning with limited multi-sentence inference and that current systems are near a performance ceiling for unambiguous cases.