[Paper Review] Generation-Augmented Retrieval for Open-domain Question Answering

Yuning Mao, Pengcheng He+5 more|arXiv (Cornell University)|Sep 17, 2020

Topic ModelingComputer Science39 references32 citations

TL;DR

Generation-Augmented Retrieval (GAR) expands queries with generated contexts from a language model, improving open-domain retrieval performance and end-to-end QA when combined with sparse BM25, and is complementary to dense retrievers like DPR.

ABSTRACT

We propose Generation-Augmented Retrieval (GAR) for answering open-domain questions, which augments a query through text generation of heuristically discovered relevant contexts without external resources as supervision. We demonstrate that the generated contexts substantially enrich the semantics of the queries and GAR with sparse representations (BM25) achieves comparable or better performance than state-of-the-art dense retrieval methods such as DPR. We show that generating diverse contexts for a query is beneficial as fusing their results consistently yields better retrieval accuracy. Moreover, as sparse and dense representations are often complementary, GAR can be easily combined with DPR to achieve even better performance. GAR achieves state-of-the-art performance on Natural Questions and TriviaQA datasets under the extractive QA setup when equipped with an extractive reader, and consistently outperforms other retrieval methods when the same generative reader is used.

Motivation & Objective

• Motivate improving OpenQA retriever performance by enriching queries with generated contexts without external supervision.
• Propose GAR to generate multiple generation targets (answer, sentence, and title) to augment queries.
• Evaluate GAR with sparse BM25 and with readers on Natural Questions and TriviaQA datasets.
• Show that diverse generation targets improve retrieval and that GAR can be combined with dense retrievers for further gains.

Proposed method

• Augment a question by generating contexts using a pre-trained language model (BART-large) for three targets: answer, sentence containing the answer, and title of the passage.
• Append generated contexts to the original question to form generation-augmented queries.
• Retrieval is performed with BM25 (sparse) and results from multiple augmented queries are fused equally or via simple fusion methods.
• Use an extractive reader (DPR-like) and a generative reader (sequence-to-sequence with BART) to evaluate end-to-end QA performance.
• Evaluate on Natural Questions and TriviaQA with top-k retrieval accuracy and Exact Match (EM) as QA metrics.
• Demonstrate that Gar with sparse representations can rival or exceed dense methods like DPR and that Gar+ can outperform either individually.

Experimental results

Research questions

• RQ1 Does generation-augmented retrieval improve passage retrieval accuracy over BM25 and over DPR alone?
• RQ2 How do different generation targets (answer, sentence, title) contribute to retrieval, and is fusion beneficial?
• RQ3 Can GAR complement dense retrievers to further improve end-to-end QA performance?
• RQ4 What is the end-to-end QA performance of GAR in extractive and generative reading setups on NQ and TriviaQA?

Key findings

• GAR with BM25 achieves comparable or better retrieval performance than dense methods on NQ and TriviaQA for many k values.
• Using multiple generation targets and fusing results yields robust gains across datasets.
• Gar+ (GAR fused with DPR) consistently improves retrieval over either method alone, achieving state-of-the-art end-to-end results in extractive OpenQA.
• End-to-end Extractive QA: EM=41.8 (43.8 with Gar+) on NQ and 62.7 on Trivia; Generative QA: EM=38.1 (45.3 with Gar+) on NQ and 62.2 on Trivia.
• GAR demonstrates strong efficiency, with generation-augmented queries adding modest overhead yet delivering substantial retrieval gains compared to baselines.