[Paper Review] Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets

Saku Sugawara, Pontus Stenetorp+2 more|arXiv (Cornell University)|Nov 21, 2019

Topic Modeling34 references4 citations

TL;DR

This paper proposes a semi-automated, ablation-based methodology to assess whether machine reading comprehension (MRC) datasets truly benchmark advanced language understanding skills. By systematically removing features tied to specific linguistic skills—such as content words, sentence order, or POS tags—it shows that strong baseline models achieve 89.2% and 78.5% of original performance on content-only and shuffled-word versions, indicating most existing MRC questions rely on surface-level cues rather than deep comprehension.

ABSTRACT

Existing analysis work in machine reading comprehension (MRC) is largely concerned with evaluating the capabilities of systems. However, the capabilities of datasets are not assessed for benchmarking language understanding precisely. We propose a semi-automated, ablation-based methodology for this challenge; By checking whether questions can be solved even after removing features associated with a skill requisite for language understanding, we evaluate to what degree the questions do not require the skill. Experiments on 10 datasets (e.g., CoQA, SQuAD v2.0, and RACE) with a strong baseline model show that, for example, the relative scores of a baseline model provided with content words only and with shuffled sentence words in the context are on average 89.2% and 78.5% of the original score, respectively. These results suggest that most of the questions already answered correctly by the model do not necessarily require grammatical and complex reasoning. For precise benchmarking, MRC datasets will need to take extra care in their design to ensure that questions can correctly evaluate the intended skills.

Motivation & Objective

• To address the lack of systematic evaluation of MRC datasets' ability to benchmark true language understanding skills.
• To investigate whether existing MRC questions require complex reasoning or grammatical understanding, or can be solved via superficial patterns.
• To develop a scalable, semi-automated method for assessing the benchmarking capacity of MRC datasets based on requisite skills.
• To identify design flaws in current datasets that allow models to succeed without mastering intended NLU capabilities.

Proposed method

• Proposes a semi-automated ablation-based methodology where input text and questions are systematically modified to remove features associated with specific linguistic skills.
• Defines 12 requisite skills—grouped into reading and reasoning categories—such as coreference resolution, temporal reasoning, and lexical inference.
• Applies input ablation techniques including vocabulary anonymization, sentence shuffling, and part-of-speech (POS) tagging to test model robustness.
• Uses a strong baseline model (e.g., BERT-based) to evaluate performance on ablated inputs, comparing relative scores to original performance.
• Employs human annotation to assess reconstructability of ablated features and their necessity for correct answers, distinguishing between cases where features are guessable or essential.
• Analyzes 10 existing MRC datasets (e.g., CoQA, SQuAD v2.0, RACE) across two answering styles: extractive and multiple-choice.

Experimental results

Research questions

• RQ1 To what extent do existing MRC datasets require deep linguistic understanding, such as coreference resolution or causal reasoning, to answer questions correctly?
• RQ2 Can strong baseline models solve MRC questions after removing content words, sentence order, or POS tags, indicating reliance on surface-level patterns?
• RQ3 Are ablated features (e.g., function words, word order) reconstructable by humans, and are they truly necessary for correct answers?
• RQ4 How do the relative performance scores on ablated inputs reflect the benchmarking capacity of MRC datasets for evaluating complex NLU skills?
• RQ5 What design principles should be followed to ensure MRC datasets properly evaluate intended language understanding capabilities?

Key findings

• On average, the baseline model achieved 89.2% of its original F1 score when only content words were retained in the context and question, indicating strong performance on surface-level cues.
• When sentence words were shuffled, the model’s performance dropped to 78.5% of the original score, suggesting that word order and syntactic structure are not always essential for correct answers.
• Performance on shuffled sentence order was 95.4% of the original score, indicating that sentence-level syntactic structure is often not critical for model success.
• Human annotation revealed that for most questions, ablated features (e.g., function words) were not reconstructable, and even when guessable, were not necessary for correct answers.
• No questions were found in Case δ (reconstructable and necessary features), suggesting that most questions do not require deep linguistic understanding to be solved.
• The results collectively indicate that many existing MRC questions are insufficient for evaluating complex language understanding, as models can succeed via pattern matching rather than true comprehension.