[Paper Review] Towards Understanding Retrieval Accuracy and Prompt Quality in RAG Systems

Shaohua Zhao, Yuheng Huang+4 more|arXiv (Cornell University)|Nov 29, 2024

Fault Detection and Control Systems4 citations

TL;DR

This paper presents a systematic empirical study of three core RAG deployment decisions—whether to use RAG, how much retrieval to perform, and how to integrate retrieved knowledge—across three LLMs and six datasets. It reveals that RAG success is highly task- and model-dependent, with optimal retrieval volume varying between 5–10 documents for QA and requiring scenario-specific tuning for code generation, while prompting methods significantly improve code generation but offer minimal gains in question answering.

ABSTRACT

Retrieval-Augmented Generation (RAG) is a pivotal technique for enhancing the capability of large language models (LLMs) and has demonstrated promising efficacy across a diverse spectrum of tasks. While LLM-driven RAG systems show superior performance, they face unique challenges in stability and reliability. Their complexity hinders developers' efforts to design, maintain, and optimize effective RAG systems. Therefore, it is crucial to understand how RAG's performance is impacted by its design. In this work, we conduct an early exploratory study toward a better understanding of the mechanism of RAG systems, covering three code datasets, three QA datasets, and two LLMs. We focus on four design factors: retrieval document type, retrieval recall, document selection, and prompt techniques. Our study uncovers how each factor impacts system correctness and confidence, providing valuable insights for developing an accurate and reliable RAG system. Based on these findings, we present nine actionable guidelines for detecting defects and optimizing the performance of RAG systems. We hope our early exploration can inspire further advancements in engineering, improving and maintaining LLM-driven intelligent software systems for greater efficiency and reliability.

Motivation & Objective

• To identify and analyze the fundamental engineering trade-offs in Retrieval-Augmented Generation (RAG) systems that impact real-world performance.
• To investigate whether universal RAG strategies are effective across diverse tasks and models.
• To provide evidence-based guidance for practitioners on when and how to deploy RAG effectively.
• To understand how retrieval volume and prompting strategies influence downstream task accuracy in LLMs.

Proposed method

• Conducted systematic experiments across three large language models (LLMs) on six datasets spanning question answering and code generation tasks.
• Evaluated three core RAG design decisions: deployment selection, retrieval quantity, and knowledge integration via prompting.
• Measured retrieval accuracy and prompt quality using task-specific metrics across varied model and task configurations.
• Used controlled ablation studies to isolate the impact of retrieval volume and prompting techniques on model outputs.
• Applied variable recall thresholds to assess failure modes and robustness under imperfect retrieval.
• Analyzed task-specific patterns in optimal retrieval volume and integration effectiveness using quantitative performance metrics.

Experimental results

Research questions

• RQ1 When should RAG be deployed, and what factors determine its success or failure on a per-sample basis?
• RQ2 What is the optimal number of retrieved documents for different tasks, and does this vary across question answering and code generation?
• RQ3 How does the choice of prompting strategy affect knowledge integration effectiveness in RAG systems?
• RQ4 To what extent do retrieval accuracy and prompt quality jointly influence final output quality in RAG pipelines?
• RQ5 Are there universal RAG configurations that work across diverse tasks, or is context-aware design essential?

Key findings

• RAG deployment must be highly selective, as up to 12.6% of samples fail even with perfect retrieved documents due to task-specific failure modes.
• For question answering tasks, an optimal retrieval volume of 5–10 documents consistently outperforms other settings across models and datasets.
• Code generation tasks require scenario-specific optimization of retrieval volume, as no universal optimal range was observed.
• Prompting methods significantly improve knowledge integration in code generation, but show minimal impact in question answering tasks.
• Knowledge integration effectiveness is strongly dependent on task characteristics and model capabilities, undermining the viability of universal RAG strategies.
• The study demonstrates that effective RAG systems require context-aware design decisions rather than one-size-fits-all configurations.