[Paper Review] ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension

Sheng Zhang, Xiaodong Liu+4 more|arXiv (Cornell University)|Oct 30, 2018

Topic Modeling30 references215 citations

TL;DR

The paper introduces ReCoRD, a large-scale MRC dataset requiring commonsense reasoning, showing that humans markedly outperform state-of-the-art models and highlighting the gap to bridge.

ABSTRACT

We present a large-scale dataset, ReCoRD, for machine reading comprehension requiring commonsense reasoning. Experiments on this dataset demonstrate that the performance of state-of-the-art MRC systems fall far behind human performance. ReCoRD represents a challenge for future research to bridge the gap between human and machine commonsense reading comprehension. ReCoRD is available at http://nlp.jhu.edu/record.

Motivation & Objective

• Motivate the need for reading comprehension requiring broad commonsense reasoning beyond surface-level text patterns.
• Automatically generate a large benchmark (passage, cloze-style query, answer) from news articles to assess commonsense reasoning.
• Apply filtering and human validation to ensure questions require non-trivial reasoning and are unambiguous.
• Provide baselines and human performance to quantify the gap between machines and humans on commonsense MRC.

Proposed method

• Automatically generate 770k (passage, query, answer) triples from CNN/Daily Mail news articles.
• Form cloze-style queries by replacing a named entity with X in sentences that cite antecedents in the passage.
• Filter easy triples using a strong MRC model (SAN) to keep 244k harder triples.
• Crowdsource human validation to prune ambiguity and ensure correct answers, yielding a 120,730 query set across train/dev/test splits.
• Evaluate multiple MRC models (including DocQA with/without ELMo, QANet, ASReader, SAN, language models) and human performance on exact match and F1 metrics.

Experimental results

Research questions

• RQ1 How do current MRC models perform on a dataset that requires commonsense reasoning?
• RQ2 What is the performance gap between humans and machines on ReCoRD across standard MRC architectures?
• RQ3 What types of commonsense reasoning are most prevalent in ReCoRD and how do models fare on them?
• RQ4 Does candidate-entity guidance (the cloze setting) help models, and how does data construction affect difficulty?

Key findings

• Humans achieve 91.31 EM and 91.69 F1 on the test set, while the best automatic method (DocQA with ELMo) achieves 46.65 F1 and 45.44 EM on the test set.
• SAN-based filtering confirms that many queries are hard across models, with substantially lower scores than humans.
• Unsupervised language models perform similarly to random guessing on ReCoRD, suggesting domain knowledge gaps.
• Eliciting answers from candidate entities (cloze setting) provides potential gains (~6% OOC reduction) if models leverage entity candidates.
• Across 100 sampled queries, 75% require commonsense reasoning, with major types including conceptual knowledge and causal/naïve psychology reasoning.