[Paper Review] Reading Comprehension as Natural Language Inference: A Semantic Analysis

Anshuman Mishra, Dhruvesh Patel+4 more|arXiv (Cornell University)|Oct 4, 2020

Topic Modeling15 references4 citations

TL;DR

This paper investigates reading comprehension as natural language inference (NLI) by converting the RACE dataset into an NLI format and comparing RoBERTa models trained on QA versus NLI forms. It finds that NLI-based models outperform QA-based models on deductive, dialogue, and math reasoning questions, while QA models perform better on negation questions, highlighting task-form-dependent performance advantages.

ABSTRACT

In the recent past, Natural language Inference (NLI) has gained significant attention, particularly given its promise for downstream NLP tasks. However, its true impact is limited and has not been well studied. Therefore, in this paper, we explore the utility of NLI for one of the most prominent downstream tasks, viz. Question Answering (QA). We transform the one of the largest available MRC dataset (RACE) to an NLI form, and compare the performances of a state-of-the-art model (RoBERTa) on both these forms. We propose new characterizations of questions, and evaluate the performance of QA and NLI models on these categories. We highlight clear categories for which the model is able to perform better when the data is presented in a coherent entailment form, and a structured question-answer concatenation form, respectively.

Motivation & Objective

• To evaluate whether transforming reading comprehension into a natural language inference (NLI) task improves model performance.
• To identify semantic reasoning categories where NLI-based models outperform traditional QA models.
• To analyze the impact of data format—question-answer concatenation versus premise-hypothesis pairs—on model performance.
• To provide a semantic characterization of question types that benefit from NLI formulation.
• To guide model selection based on question type and reasoning category in reading comprehension tasks.

Proposed method

• Converted the RACE dataset (excluding cloze-style questions) into an NLI format using rule-based methods, transforming each question-answer pair into a hypothesis with the passage as the premise.
• Applied dependency parsing and paraphrasing rules from prior work (Demszky et al., 2018) with custom extensions for RACE-specific question types, such as 'which of the following are not true'.
• Fine-tuned RoBERTa on both the original QA-formatted RACE data and the converted NLI-formatted data to compare model performance.
• Conducted manual annotation of 175 dev-set examples (from a delta subset where QA and NLI models disagreed) into 7 semantic reasoning categories: linguistic matching, main idea, negation, dialogue, math, deductive, and inductive reasoning.
• Used heuristic-based categorization (e.g., keywords like 'not', 'how many', 'true') to define non-exclusive question types for quantitative comparison.
• Evaluated model performance across both manual and heuristic categories to identify form-dependent advantages.

Experimental results

Research questions

• RQ1 In which semantic reasoning categories does an NLI-based model outperform a QA-based model on reading comprehension tasks?
• RQ2 How does the performance of RoBERTa differ when trained on QA-formatted versus NLI-formatted reading comprehension data?
• RQ3 To what extent does proper hypothesis generation in NLI conversion improve model performance compared to simple question-answer concatenation?
• RQ4 Are there specific question types—such as those involving dialogue, negation, or mathematical reasoning—where the NLI formulation yields significant performance gains?
• RQ5 Does the NLI formulation provide advantages even without pretraining on NLI datasets, particularly for complex reasoning tasks?

Key findings

• The NLI-based model outperformed the QA-based model on dialogue reasoning questions, achieving 83.60% accuracy compared to 80.65% for the QA model.
• For deductive reasoning questions (e.g., 'which statement is true?'), the NLI model achieved 88.29% accuracy, significantly outperforming the QA model’s 81.91%.
• In math reasoning questions (e.g., 'how many functions of snow are discussed?'), the NLI model achieved 55.00% accuracy versus 45.00% for the QA model, showing a clear advantage.
• The QA model performed better on negation questions (e.g., 'which is NOT true?'), achieving 80.86% accuracy compared to the NLI model’s 77.77%, indicating a preference for QA form in this category.
• The NLI model showed consistent gains on questions requiring inference from dialogue or precise textual deduction, suggesting that structured hypothesis generation enhances reasoning in these cases.
• Manual analysis revealed that 66% of the loss cases (where QA outperformed NLI) were due to poor hypothesis generation in complex question types, highlighting the importance of high-quality conversion rules.