[Paper Review] RAGProbe: An Automated Approach for Evaluating RAG Applications

Shangeetha Sivasothy, Scott Barnett+3 more|arXiv (Cornell University)|Sep 24, 2024

Cardiovascular Function and Risk FactorsMedicine3 citations

TL;DR

RAGProbe is an automated framework that generates diverse, scenario-based question-answer pairs to evaluate Retrieval-Augmented Generation (RAG) pipelines. By introducing six evaluation scenarios targeting complex query patterns, it identifies critical failure points—especially in multi-question and multi-document queries—achieving a 51% higher average failure rate than state-of-the-art methods.

ABSTRACT

Retrieval Augmented Generation (RAG) is increasingly being used when building Generative AI applications. Evaluating these applications and RAG pipelines is mostly done manually, via a trial and error process. Automating evaluation of RAG pipelines requires overcoming challenges such as context misunderstanding, wrong format, incorrect specificity, and missing content. Prior works therefore focused on improving evaluation metrics as well as enhancing components within the pipeline using available question and answer datasets. However, they have not focused on 1) providing a schema for capturing different types of question-answer pairs or 2) creating a set of templates for generating question-answer pairs that can support automation of RAG pipeline evaluation. In this paper, we present a technique for generating variations in question-answer pairs to trigger failures in RAG pipelines. We validate 5 open-source RAG pipelines using 3 datasets. Our approach revealed the highest failure rates when prompts combine multiple questions: 91% for questions when spanning multiple documents and 78% for questions from a single document; indicating a need for developers to prioritise handling these combined questions. 60% failure rate was observed in academic domain dataset and 53% and 62% failure rates were observed in open-domain datasets. Our automated approach outperforms the existing state-of-the-art methods, by increasing the failure rate by 51% on average per dataset. Our work presents an automated approach for continuously monitoring the health of RAG pipelines, which can be integrated into existing CI/CD pipelines, allowing for improved quality.

Motivation & Objective

• To address the lack of automated, systematic evaluation for RAG pipelines, which currently rely on manual, trial-and-error testing.
• To identify and categorize variations in question-answer pairs that expose failure points in RAG systems, such as context misunderstanding and incomplete responses.
• To develop a schema and template-based approach for generating domain-specific, diverse QA pairs that stress-test RAG pipelines.
• To improve RAG pipeline quality by enabling continuous integration testing through automated, repeatable evaluation scenarios.
• To outperform existing evaluation frameworks like RAGAS by generating more effective, failure-inducing test cases.

Proposed method

• Proposes six distinct evaluation scenarios to represent diverse QA variations, including single- and multi-document queries, multiple questions in one prompt, and out-of-distribution queries.
• Designs scenario-specific prompting strategies and document sampling/chunking methods to generate realistic, challenging test cases.
• Employs LLMs to generate question-answer pairs based on document corpora, using structured templates derived from common RAG failure patterns.
• Integrates evaluation metrics such as correctness, relevance, completeness, consistency, and contradiction to assess RAG outputs.
• Validates RAGProbe across five open-source RAG pipelines (e.g., Verba, Danswer, Ragflow) using three benchmark datasets: Qasper, Google NQ, and MS Marco.
• Automates pipeline health monitoring by integrating RAGProbe into CI/CD workflows for continuous evaluation.

Figure 1. RAGProbe: Our automated approach to generate question-answer pairs. Our approach is extensible by adding different evaluation scenarios and different evaluation metrics.

Experimental results

Research questions

• RQ1 How effective are scenario-based QA pairs in exposing failure modes in RAG pipelines compared to existing evaluation methods?
• RQ2 Which types of question-answer variations—especially multi-question or multi-document queries—most frequently trigger RAG pipeline failures?
• RQ3 To what extent does RAGProbe outperform state-of-the-art evaluation frameworks like RAGAS in detecting RAG pipeline defects?
• RQ4 How does domain specificity (academic vs. open-domain) influence failure rates in RAG pipelines under automated testing?
• RQ5 Can automated, template-driven QA generation produce more valid and failure-inducing test cases than existing data generation techniques?

Key findings

• The highest failure rate (91%) occurred in scenarios combining multiple questions within a single document, indicating a critical vulnerability in RAG pipeline design.
• Scenarios involving multiple questions across multiple documents resulted in a 78% failure rate, highlighting challenges in multi-hop reasoning and context aggregation.
• RAGProbe detected a 51% higher average failure rate across datasets compared to RAGAS, demonstrating superior defect detection capability.
• Academic-domain datasets (Qasper) showed a 60% failure rate, while open-domain datasets (Google NQ and MS Marco) reported 53% and 62% failure rates, respectively.
• RAGProbe generated 90–98% valid QA pairs across datasets, outperforming RAGAS’s 85–93% validity, indicating higher data quality and test coverage.
• The evaluation scenarios revealed that prompt design and document retrieval strategies significantly impact pipeline robustness, especially under complex query patterns.

Figure 2. Total failure rate combining all 5 RAG pipelines. The failure rate is calculated as the number of failures divided by the total number of questions.