[Paper Review] Enhancing Retrieval and Managing Retrieval: A Four-Module Synergy for Improved Quality and Efficiency in RAG Systems

Yunxiao Shi, Xing Zi+4 more|arXiv (Cornell University)|Jul 15, 2024

AI-based Problem Solving and Planning4 citations

TL;DR

This paper proposes a four-module synergy—Query Rewriter+, Knowledge Filter, Memory Knowledge Reservoir, and Retrieval Trigger—to enhance retrieval quality and system efficiency in Retrieval-Augmented Generation (RAG) systems. By generating multiple intent-cleared queries, filtering irrelevant knowledge, caching historical results, and dynamically triggering retrievals, the framework improves factual accuracy and reduces redundant computation across six QA benchmarks.

ABSTRACT

Retrieval-augmented generation (RAG) techniques leverage the in-context learning capabilities of large language models (LLMs) to produce more accurate and relevant responses. Originating from the simple 'retrieve-then-read' approach, the RAG framework has evolved into a highly flexible and modular paradigm. A critical component, the Query Rewriter module, enhances knowledge retrieval by generating a search-friendly query. This method aligns input questions more closely with the knowledge base. Our research identifies opportunities to enhance the Query Rewriter module to Query Rewriter+ by generating multiple queries to overcome the Information Plateaus associated with a single query and by rewriting questions to eliminate Ambiguity, thereby clarifying the underlying intent. We also find that current RAG systems exhibit issues with Irrelevant Knowledge; to overcome this, we propose the Knowledge Filter. These two modules are both based on the instruction-tuned Gemma-2B model, which together enhance response quality. The final identified issue is Redundant Retrieval; we introduce the Memory Knowledge Reservoir and the Retriever Trigger to solve this. The former supports the dynamic expansion of the RAG system's knowledge base in a parameter-free manner, while the latter optimizes the cost for accessing external knowledge, thereby improving resource utilization and response efficiency. These four RAG modules synergistically improve the response quality and efficiency of the RAG system. The effectiveness of these modules has been validated through experiments and ablation studies across six common QA datasets. The source code can be accessed at https://github.com/Ancientshi/ERM4.

Motivation & Objective

• Address the limitations of single-query retrieval in RAG systems, which suffer from information plateaus and poor intent alignment.
• Reduce response noise and improve factual accuracy by filtering out irrelevant knowledge retrieved from external sources.
• Mitigate redundant retrieval by caching frequently accessed knowledge and dynamically managing when to retrieve external data.
• Enhance system efficiency and scalability through a parameter-free memory reservoir and a retrieval trigger mechanism.
• Integrate these modules into a modular RAG pipeline to create a flexible, high-performance QA system

Proposed method

• Propose Query Rewriter+—a fine-tuned Gemma-2B model that simultaneously generates multiple semantically diverse queries and clarifies ambiguous input questions in a single inference step.
• Implement a Knowledge Filter module based on natural language inference (NLI), also fine-tuned on Gemma-2B, to assess and filter out irrelevant retrieved passages.
• Introduce a non-parametric Memory Knowledge Reservoir that caches and retrieves historically seen question-answer pairs to avoid redundant external retrievals.
• Design a Retrieval Trigger mechanism that uses dynamic thresholds to determine when to access external knowledge, optimizing cost and latency.
• Integrate all four modules into a unified RAG pipeline, replacing the basic 'retrieve-then-read' flow with a synergistic, multi-stage process.
• Train and evaluate all components using supervised fine-tuning on curated datasets, with all models based on the instruction-tuned Gemma-2B architecture.

Experimental results

Research questions

• RQ1 Can generating multiple, diverse queries from a single input question significantly improve retrieval coverage and reduce information plateaus in RAG systems?
• RQ2 To what extent can a unified query rewriting and clarification module improve intent understanding and reduce ambiguity in user questions?
• RQ3 How effective is a fine-tuned NLI-based Knowledge Filter in removing irrelevant or noisy context from retrieved passages to improve response quality?
• RQ4 Can a parameter-free memory reservoir reduce redundant retrieval and improve response efficiency without increasing model parameters?
• RQ5 What thresholding strategy in a retrieval trigger mechanism optimally balances accuracy and efficiency in dynamic RAG systems?

Key findings

• The Query Rewriter+ module significantly improves retrieval quality by generating multiple, semantically distinct queries, increasing the likelihood of retrieving relevant answer keywords.
• The Knowledge Filter module reduces response noise by filtering out irrelevant retrieved passages, enhancing the factual accuracy and robustness of LLM-generated answers.
• The Memory Knowledge Reservoir reduces redundant external retrievals by caching and reusing historically retrieved knowledge, improving system efficiency.
• The Retrieval Trigger mechanism dynamically controls when to access external knowledge, reducing unnecessary calls and optimizing resource utilization.
• Ablation studies across six common QA datasets demonstrate that the full four-module system achieves superior response quality and efficiency compared to baseline RAG pipelines.
• The proposed framework improves factual consistency and reduces hallucinations by ensuring only relevant, high-quality information is used in generation.