[Paper Review] S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension

Chuanqi Tan, Furu Wei+4 more|arXiv (Cornell University)|Jun 15, 2017

Topic Modeling67 citations

TL;DR

S-Net proposes an extraction-then-synthesis framework for MS-MARCO, first predicting evidence spans from passages and then generating final answers via a sequence-to-sequence synthesis model that uses the evidence as features.

ABSTRACT

In this paper, we present a novel approach to machine reading comprehension for the MS-MARCO dataset. Unlike the SQuAD dataset that aims to answer a question with exact text spans in a passage, the MS-MARCO dataset defines the task as answering a question from multiple passages and the words in the answer are not necessary in the passages. We therefore develop an extraction-then-synthesis framework to synthesize answers from extraction results. Specifically, the answer extraction model is first employed to predict the most important sub-spans from the passage as evidence, and the answer synthesis model takes the evidence as additional features along with the question and passage to further elaborate the final answers. We build the answer extraction model with state-of-the-art neural networks for single passage reading comprehension, and propose an additional task of passage ranking to help answer extraction in multiple passages. The answer synthesis model is based on the sequence-to-sequence neural networks with extracted evidences as features. Experiments show that our extraction-then-synthesis method outperforms state-of-the-art methods.

Motivation & Objective

• Motivate the MS-MARCO setting where answers may come from multiple passages and need synthesis beyond exact text spans.
• Develop an extraction-then-synthesis framework that first extracts evidence spans and then synthesizes final answers.
• Introduce a multi-task learning approach that includes passage ranking to enhance evidence extraction.
• Leverage a sequence-to-sequence synthesis model that uses extracted evidences as features for final answer generation.
• Demonstrate state-of-the-art performance on MS-MARCO compared to pure extraction and several baselines.

Proposed method

• Use a Bidirectional GRU-based encoding for question and passages with character-level embeddings.
• Predict evidence snippets with a pointer network that outputs start and end positions over concatenated passages.
• Implement passage ranking as a multi-task objective to improve evidence extraction.
• Train an evidence extraction model with joint loss of evidence prediction and passage ranking.
• Synthesize the final answer with a sequence-to-sequence model that conditions on question, passage, and extracted evidence positions as features.
• Decode with beam search and apply post-processing to refine generated answers.

Experimental results

Research questions

• RQ1 How can MS-MARCO-style answers, which may come from multiple spans and even outside-passage words, be effectively produced?
• RQ2 Does combining evidence extraction with synthesis improve answer quality over pure extraction or end-to-end generation?
• RQ3 Does joint passage ranking improve extraction quality and downstream synthesis?
• RQ4 Can a seq-to-seq generator effectively utilize extracted evidences to produce coherent final answers?

Key findings

• The extraction-then-synthesis framework outperforms pure extraction baselines and several competing methods on MS-MARCO in ROUGE-L and BLEU-1.
• An ensemble of extraction models further improves ROUGE-L to 42.92 and BLEU-1 to 44.97 on the test set using extraction alone.
• The synthesis model with extracted evidences as features yields additional improvements, achieving ROUGE-L of 46.65 and BLEU-1 of 44.78 (S-Net*), near human performance on ROUGE-L.
• Multi-task learning with passage ranking improves evidence extraction and overall ROUGE-L performance.
• The approach significantly benefits questions where the answer requires synthesis from multiple evidences or even words from the question.