[Paper Review] CRAG -- Comprehensive RAG Benchmark

Xiao Yang, Kai Sun+25 more|arXiv (Cornell University)|Jun 7, 2024

Medical Imaging Techniques and Applications7 citations

TL;DR

CRAG provides a 4,409-question RAG benchmark with mock web and KG retrieval to evaluate factual QA under realistic dynamism and diversity, highlighting gaps in current RAG systems.

ABSTRACT

Retrieval-Augmented Generation (RAG) has recently emerged as a promising solution to alleviate Large Language Model (LLM)'s deficiency in lack of knowledge. Existing RAG datasets, however, do not adequately represent the diverse and dynamic nature of real-world Question Answering (QA) tasks. To bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG), a factual question answering benchmark of 4,409 question-answer pairs and mock APIs to simulate web and Knowledge Graph (KG) search. CRAG is designed to encapsulate a diverse array of questions across five domains and eight question categories, reflecting varied entity popularity from popular to long-tail, and temporal dynamisms ranging from years to seconds. Our evaluation of this benchmark highlights the gap to fully trustworthy QA. Whereas most advanced LLMs achieve <=34% accuracy on CRAG, adding RAG in a straightforward manner improves the accuracy only to 44%. State-of-the-art industry RAG solutions only answer 63% of questions without any hallucination. CRAG also reveals much lower accuracy in answering questions regarding facts with higher dynamism, lower popularity, or higher complexity, suggesting future research directions. The CRAG benchmark laid the groundwork for a KDD Cup 2024 challenge and attracted thousands of participants and submissions. We commit to maintaining CRAG to serve research communities in advancing RAG solutions and general QA solutions. CRAG is available at https://github.com/facebookresearch/CRAG/.

Motivation & Objective

• Motivate creation of a realistic, diverse, and long-lasting RAG benchmark reflecting real-world QA challenges.
• Present CRAG dataset with 4,409 questions across five domains and eight question types.
• Propose a multi-task evaluation framework distinguishing hallucinations, misses, and correct answers to enable reliable comparisons.
• Demonstrate baseline LLM and state-of-the-art RAG performance to identify gaps and directions for future research.

Proposed method

• Design three retrieval tasks: web retrieval only, KG plus web retrieval augmentation, and end-to-end RAG with larger web candidate pools.
• Create 4,409 QA pairs (KG-derived and web-derived) spanning five domains and eight question types, with diverse dynamism and popularity.
• Provide mock retrieval APIs and up to 50 web pages per question to emulate real-world search and KG access.
• Define a scoring system with four labels (perfect, acceptable, missing, incorrect) and compute a Score h that penalizes hallucinations more heavily.
• Use both human and two automatic evaluators (ChatGPT and Llama 3) to estimate accuracy, hallucination, and missing rates, reporting averages across evaluators.

Experimental results

Research questions

• RQ1 How does a straightforward RAG setup improve QA accuracy on CRAG compared to LLM-only approaches?
• RQ2 How do web retrieval, structured KG querying, and larger retrieval candidate pools affect accuracy and hallucination rates across diverse question slices?
• RQ3 Which dimensions (domain, dynamism, popularity, question type) are most challenging for current RAG systems on CRAG?
• RQ4 What gaps remain for industry SOTA RAG systems in achieving hallucination-free, fully trustworthy QA on CRAG?

Key findings

• LLM-only solutions reach up to 34% accuracy on CRAG.
• Straightforward RAG improves accuracy to up to 44%.
• Industry SOTA RAG solutions answer 63% of questions without hallucination but still show substantial room for improvement.
• Performance deteriorates for real-time/fast-changing facts, tail entities, and complex questions (set, post-processing, false premises).
• KG+web (Task 2) generally yields higher scores than web-only (Task 1), indicating structured data helps, though gains are modest.
• End-to-end RAG (Task 3) with more web pages improves recall and performance over Task 2, underscoring the importance of search ranking.
• SOTA systems show better raw scores but still experience notable hallucination rates (17-25%), highlighting trustworthiness challenges.