[Paper Review] Retrieval-Augmented Generation for Natural Language Processing: A Survey

Shangyu Wu, Ying Xiong+9 more|arXiv (Cornell University)|Jul 18, 2024

Topic Modeling13 citations

TL;DR

A comprehensive survey of retrieval-augmented generation (RAG) for NLP, detailing retrievers, fusion strategies, training, generators, applications, and future challenges, with tutorial code for representative techniques.

ABSTRACT

Large language models (LLMs) have demonstrated great success in various fields, benefiting from their huge amount of parameters that store knowledge. However, LLMs still suffer from several key issues, such as hallucination problems, knowledge update issues, and lacking domain-specific expertise. The appearance of retrieval-augmented generation (RAG), which leverages an external knowledge database to augment LLMs, makes up those drawbacks of LLMs. This paper reviews all significant techniques of RAG, especially in the retriever and the retrieval fusions. Besides, tutorial codes are provided for implementing the representative techniques in RAG. This paper further discusses the RAG update, including RAG with/without knowledge update. Then, we introduce RAG evaluation and benchmarking, as well as the application of RAG in representative NLP tasks and industrial scenarios. Finally, this paper discusses RAG's future directions and challenges for promoting this field's development.

Motivation & Objective

• Explain the motivation for using external knowledge stores to alleviate LLM hallucinations and knowledge update issues.
• Systematically review RAG components, including retrievers, datastore design, and fusion methods.
• Present training strategies for RAG with and without datastore updates.
• Survey applications of RAG in NLP tasks and industrial scenarios.
• Identify future directions and challenges to advance RAG research and practice.

Proposed method

• Describe the RAG architecture comprising retriever, generator, and retrieval fusions (query-based, latent, logits-based).
• Detail the retriever construction: chunking corpora, encoding chunks, and building vector indices and datastores.
• Categorize and explain retrieval fusion techniques: query-based (text/feature concatenation), logits-based (ensemble, calibration), and latent (attention-based, weighted addition).
• Outline generator types and how retrieval augmentation is integrated into them.
• Discuss RAG training approaches with/without datastore updates and their implications.

Figure 1. The overview of retrieval-augmented generation for natural language processing.

Experimental results

Research questions

• RQ1 What are the key components and design choices in retrieval-augmented generation for NLP?
• RQ2 How do different retriever architectures, indexing strategies, and fusion methods affect RAG performance?
• RQ3 What are effective training strategies for RAG with and without datastore updates?
• RQ4 How can RAG be applied to various NLP tasks and real-world industrial scenarios?
• RQ5 What future directions and challenges are most impactful for advancing RAG?

Key findings

• RAG combines retrievers, fusions, and generators to mitigate hallucinations and enable up-to-date domain knowledge.
• Retrievers use chunking, encoding, and ANN indexing to build a scalable datastore for retrieval.
• Fusion methods are categorized into query-based, logits-based, and latent fusions with text/feature, ensemble, calibration, and attention mechanisms.
• RAG training can be performed with or without datastore updates, each offering different advantages.
• The survey discusses applications across representative NLP tasks and industry contexts and highlights future directions and challenges.

Figure 2. Two stages of using the retriever.