[Paper Review] Using Natural Language Relations between Answer Choices for Machine Comprehension

Rajkumar Pujari, Dan Goldwasser|arXiv (Cornell University)|Dec 31, 2020

Topic Modeling25 references4 citations

TL;DR

This paper proposes a method to improve machine comprehension by leveraging natural language inference (NLI) relations—such as entailment and contradiction—between answer choices. Using a standalone QA system, an NLI model, and an ILP-based inference framework, the approach jointly re-evaluates answer choices, achieving state-of-the-art results on MultiRC and SemEval-2018 Task-11, with a 21.62% EM0 score on MultiRC using self-training data.

ABSTRACT

When evaluating an answer choice for Reading Comprehension task, other answer choices available for the question and the answers of related questions about the same paragraph often provide valuable information. In this paper, we propose a method to leverage the natural language relations between the answer choices, such as entailment and contradiction, to improve the performance of machine comprehension. We use a stand-alone question answering (QA) system to perform QA task and a Natural Language Inference (NLI) system to identify the relations between the choice pairs. Then we perform inference using an Integer Linear Programming (ILP)-based relational framework to re-evaluate the decisions made by the standalone QA system in light of the relations identified by the NLI system. We also propose a multitask learning model that learns both the tasks jointly.

Motivation & Objective

• To improve machine comprehension by modeling pragmatic consistency across answer choices and related questions.
• To address the limitation of standalone QA systems that evaluate each choice in isolation, ignoring inter-choice relationships.
• To explore whether natural language inference (entailment, contradiction, neutral) between answer choices can enhance reasoning and prediction accuracy.
• To develop a joint learning framework that integrates QA and NLI tasks for improved generalization.
• To investigate the impact of self-training on NLI models for better relation detection in answer choice pairs.

Proposed method

• A standalone question answering (QA) system assigns initial true/false labels and confidence scores to each answer choice.
• A separate Natural Language Inference (NLI) system classifies relations (entailment, contradiction, neutral) between all ordered pairs of answer choices for each question.
• An Integer Linear Programming (ILP)-based relational inference framework re-evaluates QA predictions using the NLI relations to enforce logical consistency.
• A joint multitask learning model trains the QA and NLI components together, sharing representations to improve both tasks.
• Self-training is applied to the NLI model using predictions from the MultiRC dataset to improve relation detection performance.
• The framework uses confidence scores from both QA and NLI systems as inputs to the ILP optimization process.

Experimental results

Research questions

• RQ1 Can modeling natural language inference relations between answer choices improve machine comprehension performance?
• RQ2 How effective is ILP-based inference in enforcing consistency across answer choices using NLI relations?
• RQ3 Does joint training of QA and NLI components lead to better performance than separate training?
• RQ4 To what extent does self-training on the MultiRC dataset improve NLI relation detection for answer choices?
• RQ5 Can implicit or contextual relationships beyond strict entailment/contradiction further enhance comprehension?

Key findings

• The method improved EM0 on the MultiRC dataset from 18.15% (standalone QA) to 21.62% when using a self-trained NLI model.
• The joint model achieved 20.36% EM0 and 57.08% EM1 on MultiRC, outperforming standalone QA and other baselines.
• On the SemEval-2018 Task-11 dataset, the joint model achieved 85.4% accuracy on the development set and 82.1% on the test set, surpassing the standalone TriAN-single model.
• The improvement was statistically significant, as confirmed by McNemar’s chi-squared test.
• The NLI model’s performance was limited by high lexical overlap and negation words, particularly affecting entailment detection.
• Self-training on MultiRC data revealed that non-entailment/non-contradiction relations exist in answer choices, suggesting room for modeling broader semantic relationships.