[Paper Review] A Framework for Evaluation of Machine Reading Comprehension Gold Standards

Viktor Schlegel, Marco Valentino+3 more|arXiv (Cornell University)|Mar 10, 2020

Topic Modeling42 references19 citations

TL;DR

This paper proposes a unifying framework to systematically evaluate Machine Reading Comprehension (MRC) gold standards by analyzing linguistic complexity, required reasoning, background knowledge, factual correctness, and lexical cues. The framework uses a qualitative annotation schema and approximative metrics to reveal that existing MRC datasets often lack lexical ambiguity, contain factual inconsistencies in answers, and rely heavily on superficial lexical cues, undermining true reading comprehension evaluation quality.

ABSTRACT

Machine Reading Comprehension (MRC) is the task of answering a question over a paragraph of text. While neural MRC systems gain popularity and achieve noticeable performance, issues are being raised with the methodology used to establish their performance, particularly concerning the data design of gold standards that are used to evaluate them. There is but a limited understanding of the challenges present in this data, which makes it hard to draw comparisons and formulate reliable hypotheses. As a first step towards alleviating the problem, this paper proposes a unifying framework to systematically investigate the present linguistic features, required reasoning and background knowledge and factual correctness on one hand, and the presence of lexical cues as a lower bound for the requirement of understanding on the other hand. We propose a qualitative annotation schema for the first and a set of approximative metrics for the latter. In a first application of the framework, we analyse modern MRC gold standards and present our findings: the absence of features that contribute towards lexical ambiguity, the varying factual correctness of the expected answers and the presence of lexical cues, all of which potentially lower the reading comprehension complexity and quality of the evaluation data.

Motivation & Objective

• To address the lack of systematic evaluation of MRC gold standards, which are critical for assessing model performance.
• To identify and analyze flaws in existing MRC datasets, such as lexical cues and factual inaccuracies in expected answers.
• To develop a unifying framework that enables systematic comparison of MRC evaluation data across linguistic features, reasoning types, and knowledge requirements.
• To improve the reliability of MRC evaluation by distinguishing between deep comprehension and surface-level pattern matching.
• To provide a foundation for future research to build higher-quality, more robust MRC benchmarks.

Proposed method

• Proposes a qualitative annotation schema to categorize linguistic complexity, required reasoning, background knowledge, and factual correctness in MRC gold standards.
• Introduces approximative metrics to quantify the presence of lexical cues—such as keyword overlap and proximity to answers—that may enable shallow model performance.
• Applies the framework to analyze modern MRC datasets, including HotpotQA and SQuAD, using manual and automated analysis techniques.
• Employs crowd-sourcing with an adversarial pre-trained model during annotation to filter out questions easily answered by lexical matching.
• Uses a multi-passage example (e.g., Marietta Air Force Station and Smyrna) to demonstrate how lexical cues can trivialize questions.
• Classifies questions based on whether they require inference, coreference resolution, or factual recall, and evaluates answer correctness against external knowledge.

Experimental results

Research questions

• RQ1 To what extent do existing MRC gold standards rely on lexical cues rather than true comprehension?
• RQ2 How consistent and factually correct are the expected answers in current MRC datasets?
• RQ3 What linguistic and reasoning complexities are actually present in MRC evaluation data?
• RQ4 How do the presence of background knowledge and ambiguity affect the difficulty of MRC tasks?
• RQ5 Can a unified framework systematically evaluate and compare the quality of different MRC gold standards?

Key findings

• Many MRC datasets, including HotpotQA, contain questions where the expected answer is easily locatable via unique keywords in the question and passage, reducing the need for deep comprehension.
• The expected answers in gold standards often lack factual consistency, with some answers being incorrect or based on outdated or ambiguous information.
• There is a notable absence of lexical ambiguity in the questions and passages, which limits the evaluation of models' ability to resolve ambiguous references.
• The framework reveals that a significant proportion of questions in modern MRC datasets can be solved using surface-level lexical matching rather than semantic understanding.
• The presence of adversarial pre-trained models during annotation successfully filtered out questions that could be answered by simple keyword matching, improving data quality.
• The analysis shows that current evaluation data often fails to measure true reading comprehension, as it is biased toward pattern matching and lacks sufficient reasoning and knowledge demands.