[Paper Review] Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documents

Ming Tu, Kevin Huang+4 more|arXiv (Cornell University)|Nov 1, 2019

Topic Modeling25 references18 citations

TL;DR

This paper proposes the Select, Answer and Explain (SAE) framework for interpretable multi-hop reading comprehension over multiple documents. It first filters answer-unrelated documents using a pairwise learning-to-rank document classifier, then jointly predicts answers and supporting sentences via a multi-task model with cross-task attention and graph neural networks, achieving competitive performance on HotpotQA with 39.89% joint EM and 66.45% F1.

ABSTRACT

Interpretable multi-hop reading comprehension (RC) over multiple documents is a challenging problem because it demands reasoning over multiple information sources and explaining the answer prediction by providing supporting evidences. In this paper, we propose an effective and interpretable Select, Answer and Explain (SAE) system to solve the multi-document RC problem. Our system first filters out answer-unrelated documents and thus reduce the amount of distraction information. This is achieved by a document classifier trained with a novel pairwise learning-to-rank loss. The selected answer-related documents are then input to a model to jointly predict the answer and supporting sentences. The model is optimized with a multi-task learning objective on both token level for answer prediction and sentence level for supporting sentences prediction, together with an attention-based interaction between these two tasks. Evaluated on HotpotQA, a challenging multi-hop RC data set, the proposed SAE system achieves top competitive performance in distractor setting compared to other existing systems on the leaderboard.

Motivation & Objective

• To address the challenge of interpretable multi-hop reading comprehension over multiple documents by reducing distraction from irrelevant documents.
• To improve answer prediction and supporting evidence retrieval through joint learning of answer span and supporting sentence prediction.
• To enhance model interpretability by explicitly identifying and explaining supporting sentences from multiple documents.
• To overcome limitations in existing methods, such as poor explainability and lack of document filtering, especially in high-distraction settings.
• To develop a scalable and accurate system that outperforms existing models on multi-hop QA benchmarks like HotpotQA.

Proposed method

• A document classifier trained with a novel pairwise learning-to-rank loss to filter out answer-unrelated documents and retain only relevant ones.
• A multi-task learning framework that jointly optimizes for answer span prediction at the token level and supporting sentence prediction at the sentence level.
• An attention-based interaction mechanism between the answer prediction and supporting sentence prediction heads to improve cross-task reasoning.
• A graph neural network (GNN) with three types of edges: intra-document sentence connections, inter-document sentence connections, and entity-aware connections.
• Sentence summarization using attention-based pooling to aggregate sentence-level representations for downstream prediction.
• Use of BERT-based contextual embeddings as input representations for both document filtering and answer/explanation prediction modules.

Experimental results

Research questions

• RQ1 Can a document filtering mechanism significantly reduce distraction and improve performance in multi-hop multi-document reading comprehension?
• RQ2 How effective is joint prediction of answers and supporting sentences in improving model interpretability and performance?
• RQ3 What is the contribution of different types of message passing in a graph neural network for multi-hop reasoning across documents?
• RQ4 Does incorporating attention-based interaction between answer and evidence prediction heads improve model performance?
• RQ5 How does the model perform on different reasoning types, such as bridge and comparison reasoning?

Key findings

• The full SAE model achieves 39.89% joint EM and 66.45% F1 on the HotpotQA dev set, demonstrating competitive performance against state-of-the-art systems.
• Removing the GNN module leads to a 1.4% drop in joint EM and 0.9% drop in F1, confirming the importance of graph-based message passing for reasoning.
• The intra-document sentence connection edges contribute the most to performance, while cross-document edges still provide marginal but meaningful improvements.
• Replacing attention-based sentence summarization with simple averaging reduces joint EM by nearly 2%, proving the effectiveness of attention in summarizing supporting sentences.
• The model performs better on 'comparison' reasoning types (51.18% joint EM) than on 'bridge' types (37.07% joint EM), suggesting greater difficulty in identifying bridging entities.
• Attention heatmaps visually confirm that the model correctly attends to key words like 'american' when determining nationality, validating its interpretability.