[Paper Review] Generate rather than Retrieve: Large Language Models are Strong Context Generators

Adarsh Ron|arXiv (Cornell University)|Sep 21, 2022

Topic ModelingComputer Science86 citations

TL;DR

GenRead replaces document retrieval with LLM-generated contextual documents for knowledge-intensive tasks, then reads the generated content to answer; clustering-based prompts improve coverage and achieve strong open-domain QA results without external sources.

ABSTRACT

Knowledge-intensive tasks, such as open-domain question answering (QA), require access to a large amount of world or domain knowledge. A common approach for knowledge-intensive tasks is to employ a retrieve-then-read pipeline that first retrieves a handful of relevant contextual documents from an external corpus such as Wikipedia and then predicts an answer conditioned on the retrieved documents. In this paper, we present a novel perspective for solving knowledge-intensive tasks by replacing document retrievers with large language model generators. We call our method generate-then-read (GenRead), which first prompts a large language model to generate contextutal documents based on a given question, and then reads the generated documents to produce the final answer. Furthermore, we propose a novel clustering-based prompting method that selects distinct prompts, resulting in the generated documents that cover different perspectives, leading to better recall over acceptable answers. We conduct extensive experiments on three different knowledge-intensive tasks, including open-domain QA, fact checking, and dialogue system. Notably, GenRead achieves 71.6 and 54.4 exact match scores on TriviaQA and WebQ, significantly outperforming the state-of-the-art retrieve-then-read pipeline DPR-FiD by +4.0 and +3.9, without retrieving any documents from any external knowledge source. Lastly, we demonstrate the model performance can be further improved by combining retrieval and generation. Our code and generated documents can be found at https://github.com/wyu97/GenRead.

Motivation & Objective

• Motivate knowledge-intensive tasks to reduce reliance on external document retrieval.
• Propose a generate-then-read pipeline that generates contextual documents and reads them to answer questions.
• Introduce clustering-based prompting to produce diverse generated contexts and improve recall.
• Demonstrate GenRead's effectiveness across open-domain QA, fact checking, and dialogue tasks in zero-shot and supervised settings.

Proposed method

• Propose generate-then-read (GenRead): prompt an LLM to generate contextual documents for a given question, then read the generated documents to produce the final answer.
• Use a zero-shot setting with a read step that conditions on generated documents to answer questions.
• In supervised settings, employ a FiD reader and fine-tune with generated documents; limit to 10 documents for efficiency.
• Introduce clustering-based prompting: cluster initial Q-D pairs and sample prompts from each cluster to induce diverse generated documents.
• Compare generation-based context with retrieval-based baselines (DPR, RAG, FiD) and show complementary gains when combining generated and retrieved docs.

Experimental results

Research questions

• RQ1 Can LLM-generated contextual documents replace external retrieval in knowledge-intensive tasks?
• RQ2 Does clustering-based prompting increase diversity and coverage of generated contexts?
• RQ3 How does GenRead perform in zero-shot vs. supervised settings across QA, fact checking, and dialogue?
• RQ4 Can combining generated and retrieved documents outperform either approach alone?

Key findings

• GenRead achieves 71.6 and 54.4 exact match (EM) on TriviaQA and WebQ in zero-shot, outperforming the retrieve-then-read baseline DPR-FiD without external documents.
• GenRead matches or exceeds zero-shot retrieve-then-read models on several benchmarks and, in some cases, surpasses them without using external knowledge sources.
• Clustering-based prompts improve QA performance over sampling or single prompts, increasing coverage and EM scores.
• In supervised settings, GenRead with FiD-xl achieves strong results and can outperform retrieval-based methods when combining generated and retrieved documents.
• The approach demonstrates complementary benefits: generated contexts can boost performance when merged with retrieved documents, often yielding larger gains than either approach alone.