[Paper Review] Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework

Ronak Pradeep, Nandan Thakur+4 more|arXiv (Cornell University)|Nov 14, 2024

Geophysics and Gravity Measurements6 citations

TL;DR

The paper presents an initial evaluation of the AutoNuggetizer framework for TREC 2024 RAG, showing a strong correlation between fully automatic nugget evaluations and mostly manual nugget assessments across 21 topics and 45 runs.

ABSTRACT

This report provides an initial look at partial results from the TREC 2024 Retrieval-Augmented Generation (RAG) Track. We have identified RAG evaluation as a barrier to continued progress in information access (and more broadly, natural language processing and artificial intelligence), and it is our hope that we can contribute to tackling the many challenges in this space. The central hypothesis we explore in this work is that the nugget evaluation methodology, originally developed for the TREC Question Answering Track in 2003, provides a solid foundation for evaluating RAG systems. As such, our efforts have focused on "refactoring" this methodology, specifically applying large language models to both automatically create nuggets and to automatically assign nuggets to system answers. We call this the AutoNuggetizer framework. Within the TREC setup, we are able to calibrate our fully automatic process against a manual process whereby nuggets are created by human assessors semi-manually and then assigned manually to system answers. Based on initial results across 21 topics from 45 runs, we observe a strong correlation between scores derived from a fully automatic nugget evaluation and a (mostly) manual nugget evaluation by human assessors. This suggests that our fully automatic evaluation process can be used to guide future iterations of RAG systems.

Motivation & Objective

• Assess whether a fully automatic nugget evaluation can substitute for manual nugget evaluation in RAG tracks.
• Refactor and apply the nugget evaluation methodology to RAG using large language models (LLMs).
• Calibrate automatic nugget creation and assignment against semi-manual/manual human assessments.
• Provide experimental results to guide future iterations of RAG systems.

Proposed method

• Refactor the nugget evaluation methodology from Voorhees (2003) for RAG using LLMs.
• Automatically generate nuggets from relevant documents (AutoNuggets) using GPT-4o.
• Automatically assign nuggets to system answers (AutoAssign) with a listwise LLM approach.
• Compare fully automatic Nugget Evaluation against (mostly) manual human assessments using NIST assessors.
• Evaluate across 21 topics using 45 runs to assess correlation between automatic and manual evaluations.

Experimental results

Research questions

• RQ1 Does the fully automatic nugget evaluation align with manual nugget assessments by human evaluators?
• RQ2 Can AutoNuggetizer provide a reliable automatic evaluation to guide future RAG system iterations?
• RQ3 How do automatic Nugget Creation and Assignment variants perform relative to semi-manual approaches?

Key findings

• Initial results show a strong correlation between scores from fully automatic nugget evaluation and the (mostly) manual nugget evaluation by human assessors.
• The study analyzes 21 topics across 45 runs to draw its conclusions about the automatic evaluation approach.
• The AutoNuggetizer framework leverages LLMs to both create nuggets and assign them to system answers, enabling an end-to-end automatic evaluation flow.
• Calibration is performed by comparing automatic nugget creation/assignment against semi-manual/manual processes within the TREC setup.