[Paper Review] Don't Use LLMs to Make Relevance Judgments

Ian Soboroff|arXiv (Cornell University)|Sep 23, 2024

Artificial Intelligence in Law4 citations

TL;DR

This paper argues against using large language models (LLMs) to generate relevance judgments for TREC-style information retrieval evaluations. It demonstrates that such LLM-generated judgments create a performance ceiling, preventing measurement of systems that outperform the model, and lead to flawed evaluations due to self-referential bias and model collapse.

ABSTRACT

Making the relevance judgments for a TREC-style test collection can be complex and expensive. A typical TREC track usually involves a team of six contractors working for 2-4 weeks. Those contractors need to be trained and monitored. Software has to be written to support recording relevance judgments correctly and efficiently. The recent advent of large language models that produce astoundingly human-like flowing text output in response to a natural language prompt has inspired IR researchers to wonder how those models might be used in the relevance judgment collection process. At the ACM SIGIR 2024 conference, a workshop ``LLM4Eval'' provided a venue for this work, and featured a data challenge activity where participants reproduced TREC deep learning track judgments, as was done by Thomas et al (arXiv:2408.08896, arXiv:2309.10621). I was asked to give a keynote at the workshop, and this paper presents that keynote in article form. The bottom-line-up-front message is, don't use LLMs to create relevance judgments for TREC-style evaluations.

Motivation & Objective

• To challenge the growing trend of using LLMs to generate relevance judgments for TREC-style evaluations.
• To expose the fundamental flaw in using LLMs as the source of truth in IR evaluation, where the model's output becomes the ground truth.
• To demonstrate that such an approach limits the evaluation to systems that perform worse than the LLM, creating a performance ceiling.
• To argue that using LLMs to generate relevance judgments leads to model collapse in evaluation, analogous to distributional collapse in self-supervised learning.
• To advocate for alternative uses of LLMs in evaluation that do not involve generating ground truth, such as quality control or supporting user studies.

Proposed method

• Analyzing the implications of using LLMs to produce relevance judgments, treating the model output as the ground truth for evaluation.
• Drawing parallels between LLM-based relevance judgment generation and the concept of model collapse in self-supervised learning, where performance degrades due to feedback from the model's own outputs.
• Contrasting this approach with established evaluation practices such as pooling, where human assessors evaluate documents from multiple systems to create a more robust and comprehensive relevance pool.
• Proposing that LLMs can be used to support evaluation without generating ground truth—e.g., for detecting errors in human judgments or assisting in user study design.
• Highlighting that when LLMs are fine-tuned on actual human relevance judgments, they can serve as privileged evaluators, enabling fairer measurement of system performance.
• Emphasizing that evaluation should not rely on LLMs to define truth, but rather on human-annotated relevance data, even when enhanced by LLMs in auxiliary roles.

Figure 1: Sample result from [ 12 ] , TREC-8, TREC-style pooling to depth 100.

Experimental results

Research questions

• RQ1 What are the consequences of using LLMs to generate relevance judgments in IR evaluation?
• RQ2 Why does using an LLM as the source of truth in evaluation create a performance ceiling for measured systems?
• RQ3 How does LLM-based relevance judgment generation lead to model collapse in the evaluation process?
• RQ4 Can LLMs still be useful in IR evaluation if they are not used to generate ground truth?
• RQ5 What alternative roles can LLMs play in supporting the evaluation pipeline without compromising evaluation integrity?

Key findings

• Using LLMs to generate relevance judgments creates a performance ceiling, as systems cannot be evaluated for performance exceeding the LLM’s own output.
• When the LLM used to generate relevance judgments is also part of the system being evaluated, the system will appear to underperform, even if it improves on the LLM’s output.
• Newer LLMs will always appear to perform worse than older ones in such evaluations, simply because they retrieve previously unjudged or incorrectly judged documents.
• This evaluation setup leads to a form of model collapse, where the evaluation metric degrades due to feedback from the model’s own outputs.
• LLMs can still be useful in evaluation if used to support human assessors—e.g., for error detection or automating user study components—provided they do not generate the ground truth.
• Fine-tuning LLMs on human relevance judgments allows them to serve as privileged evaluators, enabling fairer measurement of systems without the flaws of self-referential judgment generation.