[Paper Review] ARAGOG: Advanced RAG Output Grading

Matouš Eibich, Shivay Nagpal+1 more|arXiv (Cornell University)|Apr 1, 2024

Medical Imaging Techniques and Applications6 citations

TL;DR

The paper benchmarks multiple retrieval-augmented generation (RAG) techniques on retrieval precision and answer similarity, finding HyDE and LLM-based reranking boost retrieval quality, while Sentence Window retrieval yields the highest retrieval precision, with some techniques not outperforming Naive RAG.

ABSTRACT

Retrieval-Augmented Generation (RAG) is essential for integrating external knowledge into Large Language Model (LLM) outputs. While the literature on RAG is growing, it primarily focuses on systematic reviews and comparisons of new state-of-the-art (SoTA) techniques against their predecessors, with a gap in extensive experimental comparisons. This study begins to address this gap by assessing various RAG methods' impacts on retrieval precision and answer similarity. We found that Hypothetical Document Embedding (HyDE) and LLM reranking significantly enhance retrieval precision. However, Maximal Marginal Relevance (MMR) and Cohere rerank did not exhibit notable advantages over a baseline Naive RAG system, and Multi-query approaches underperformed. Sentence Window Retrieval emerged as the most effective for retrieval precision, despite its variable performance on answer similarity. The study confirms the potential of the Document Summary Index as a competent retrieval approach. All resources related to this research are publicly accessible for further investigation through our GitHub repository ARAGOG (https://github.com/predlico/ARAGOG). We welcome the community to further this exploratory study in RAG systems.

Motivation & Objective

• Assess how a spectrum of advanced RAG techniques impacts retrieval precision and answer similarity.
• Identify which combinations of retrieval and generation components yield the strongest performance across a RAG pipeline.
• Provide practical guidance on the trade-offs between retrieval quality, latency, and cost.
• Make experimental pipelines publicly available to enable replication and extension by the community.

Proposed method

• Evaluate a broad set of RAG techniques including Sentence-window retrieval, Document Summary Index, HyDE, Multi-query, MMR, Cohere rerank, and LLM rerank.
• Construct vector databases using different chunking strategies (512-token tokens with 50-token overlap, 3-sentence windows, 3072-token summaries) to reflect each method’s needs.
• Use GPT-3.5-turbo as the generator and LLM for evaluation via Tonic Validate metrics.
• Measure Retrieval Precision and Answer Similarity (0-5) across 10 runs per technique to mitigate LLM variability.
• Apply ANOVA and Tukey’s HSD for statistical significance of retrieval precision differences.

Figure 1: A high-level overview of the workflow within a Retrieval-Augmented Generation (RAG) system. This process diagram shows how a user query is processed by the system to retrieve relevant documents from a database and how these documents inform the generation of a response.

Experimental results

Research questions

• RQ1 Which RAG techniques most improve retrieval precision relative to Naive RAG?
• RQ2 How do RAG techniques trade off retrieval precision against answer similarity and cost?
• RQ3 Do techniques like HyDE or rerankers consistently outperform baseline across datasets and chunking schemes?
• RQ4 Is Sentence Window retrieval consistently superior for retrieval precision across configurations?

Key findings

• HyDE and LLM reranking significantly improve retrieval precision over Naive RAG.
• Sentence Window Retrieval yields high retrieval precision, often outperforming classic vector database baselines.
• MMR and Cohere rerank provide little to no improvement over Naive RAG in retrieval precision.
• Multi-query approaches underperform compared to baseline Naive RAG in retrieval precision.
• Document Summary Index approaches perform comparably to the best Classic VDB settings but require upfront summarization work.

Figure 2: The process flow of Hypothetical Document Embedding (HyDE) technique within a Retrieval-Augmented Generation system. The diagram illustrates the steps from the initial query input to the generation of a hypothetical answer and its use in retrieving relevant documents to inform the final ge