[Paper Review] ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems

Jon Saad-Falcon, Omar Khattab+2 more|arXiv (Cornell University)|Nov 16, 2023

Power Systems and Technologies4 citations

TL;DR

ARES is an automated evaluation framework for retrieval-augmented generation (RAG) systems that uses synthetic data and fine-tuned lightweight LLM judges to assess context relevance, answer faithfulness, and answer relevance. It achieves high accuracy with only 150 human-annotated datapoints by combining few-shot prompting, contrastive fine-tuning, and prediction-powered inference (PPI), outperforming RAGAS by 59.29 and 14.4 percentage points on average in context and answer relevance evaluation.

ABSTRACT

Evaluating retrieval-augmented generation (RAG) systems traditionally relies on hand annotations for input queries, passages to retrieve, and responses to generate. We introduce ARES, an Automated RAG Evaluation System, for evaluating RAG systems along the dimensions of context relevance, answer faithfulness, and answer relevance. By creating its own synthetic training data, ARES finetunes lightweight LM judges to assess the quality of individual RAG components. To mitigate potential prediction errors, ARES utilizes a small set of human-annotated datapoints for prediction-powered inference (PPI). Across eight different knowledge-intensive tasks in KILT, SuperGLUE, and AIS, ARES accurately evaluates RAG systems while using only a few hundred human annotations during evaluation. Furthermore, ARES judges remain effective across domain shifts, proving accurate even after changing the type of queries and/or documents used in the evaluated RAG systems. We make our code and datasets publicly available on Github.

Motivation & Objective

• To address the high cost and low scalability of human-annotated evaluation for RAG systems.
• To develop an automated, data-efficient evaluation framework that maintains high accuracy across domain shifts.
• To enable precise, statistically grounded scoring of RAG components using lightweight LLM judges and minimal human supervision.
• To reduce reliance on hand-crafted prompts by tailoring LLM judges to specific RAG pipelines through synthetic data.
• To provide confidence intervals for evaluation scores via prediction-powered inference (PPI), enhancing reliability.

Proposed method

• ARES generates synthetic question-answer pairs from in-domain passages using a language model, enabling data-efficient training of specialized LLM judges.
• It fine-tunes three lightweight LLM judges using a contrastive learning objective to classify context relevance, answer faithfulness, and answer relevance.
• Prediction-powered inference (PPI) combines high-accuracy LLM predictions with a small set of 150 or more human-annotated datapoints to produce confidence intervals for evaluation scores.
• Few-shot examples of in-domain queries and answers are used to prompt the LLM during synthetic data generation, ensuring domain alignment.
• The framework leverages DeBERTa-v3-Large as the base model for fine-tuned judges and validates performance across KILT and SuperGLUE benchmarks.
• GPT-4 is used to generate synthetic labels as a low-cost alternative to human annotation, with performance improving as more labels are generated.

Figure 1: Overview of ARES : As inputs, the ARES pipeline requires an in-domain passage set, a human preference validation set of 150 annotated datapoints or more, and five few-shot examples of in-domain queries and answers, which are used for prompting LLMs in synthetic data generation. To prepare

Experimental results

Research questions

• RQ1 Can an automated RAG evaluation framework achieve high accuracy with minimal human annotations, specifically fewer than 200 human-labeled examples?
• RQ2 How effective is synthetic data generation combined with contrastive fine-tuning in training specialized LLM judges for RAG evaluation?
• RQ3 To what extent does prediction-powered inference (PPI) improve the reliability and precision of LLM-based evaluation scores?
• RQ4 Can GPT-4-generated labels substitute for human-annotated preference data while maintaining evaluation accuracy?
• RQ5 How robust are ARES judges to domain shifts when evaluating RAG systems on out-of-domain queries or documents?

Key findings

• ARES outperforms RAGAS by 59.29 percentage points on average in context relevance evaluation accuracy across six KILT and SuperGLUE datasets.
• ARES achieves 14.4 percentage point higher average accuracy than RAGAS in answer relevance evaluation across the same benchmarks.
• On the MultiRC dataset, ARES achieves a Kendall’s tau of 0.28 for context relevance and 0.28 for answer relevance, indicating strong alignment with human preferences.
• When using GPT-4-generated labels instead of human annotations, ARES maintains a Kendall’s tau of 0.78 for context relevance and 0.72 for answer relevance on NQ, showing strong performance with minimal human input.
• The PPI confidence interval range averages 8.3% across datasets, demonstrating reliable statistical bounds for evaluation scores.
• ARES maintains consistent performance across domain shifts, including when evaluating on T-Rex (KILT) and CodeSearchNet, with Kendall’s tau of 0.38 and 0.28 respectively, showing generalization capability.