[Paper Review] LLMs Can Patch Up Missing Relevance Judgments in Evaluation

Shivani Upadhyay, Ehsan Kamalloo+1 more|arXiv (Cornell University)|May 8, 2024

Artificial Intelligence in Law4 citations

TL;DR

This paper proposes a framework that uses large language models (LLMs) to automatically predict fine-grained relevance judgments for unjudged documents ("holes") in information retrieval benchmarks. By prompting LLMs with detailed instructions and few-shot examples, the method achieves high correlation (Kendall τ of 0.87–0.92) with human-annotated relevance judgments even when only 10% of ground-truth judgments are retained, significantly outperforming simple non-relevant labeling.

ABSTRACT

Unjudged documents or holes in information retrieval benchmarks are considered non-relevant in evaluation, yielding no gains in measuring effectiveness. However, these missing judgments may inadvertently introduce biases into the evaluation as their prevalence for a retrieval model is heavily contingent on the pooling process. Thus, filling holes becomes crucial in ensuring reliable and accurate evaluation. Collecting human judgment for all documents is cumbersome and impractical. In this paper, we aim at leveraging large language models (LLMs) to automatically label unjudged documents. Our goal is to instruct an LLM using detailed instructions to assign fine-grained relevance judgments to holes. To this end, we systematically simulate scenarios with varying degrees of holes by randomly dropping relevant documents from the relevance judgment in TREC DL tracks. Our experiments reveal a strong correlation between our LLM-based method and ground-truth relevance judgments. Based on our simulation experiments conducted on three TREC DL datasets, in the extreme scenario of retaining only 10% of judgments, our method achieves a Kendall tau correlation of 0.87 and 0.92 on an average for Vicuña-7B and GPT-3.5 Turbo respectively.

Motivation & Objective

• To address the bias introduced by incomplete relevance judgments in IR test collections, where unjudged documents are treated as non-relevant, skewing evaluation metrics.
• To reduce reliance on costly and time-consuming human labeling by automating the prediction of relevance labels for unjudged documents using LLMs.
• To develop a robust, instruction-based LLM framework capable of assigning fine-grained relevance judgments (e.g., TREC-style labels) rather than binary labels.
• To evaluate the effectiveness of LLMs across varying degrees of missing judgments, simulating realistic benchmark holes.
• To ensure the method is free from data leakage by testing on unseen datasets like TREC DL 2023.

Proposed method

• The authors simulate incomplete test collections by randomly removing 90% of gold relevance judgments from TREC DL 2019, 2020, and 2021 datasets, creating 'holes' in the judgment data.
• A prompt engineering strategy is used to instruct LLMs (Vicuña-7B and GPT-3.5 Turbo) to assign fine-grained relevance labels (e.g., 0 to 3) to unjudged query-document pairs based on detailed instructions and few-shot examples.
• The LLM assessor is prompted with a structured prompt that includes the query, document text, and a clear definition of relevance levels per TREC guidelines.
• The framework evaluates the correlation between LLM-predicted judgments and ground-truth judgments using Kendall’s τ, a standard metric for ranking agreement.
• The method is tested under both zero-shot and few-shot prompting settings to compare performance and variance.
• Data contamination is tested by applying the LLM to TREC DL 2023 qrels, which were not seen during training, to rule out data leakage.

Experimental results

Research questions

• RQ1 Can LLMs accurately predict fine-grained relevance judgments for unjudged documents in IR test collections?
• RQ2 How does the performance of LLM-based judgment prediction vary with increasing degrees of missing relevance judgments?
• RQ3 Does few-shot prompting outperform zero-shot prompting in terms of correlation with human judgments and stability?
• RQ4 Can LLMs generalize to unseen datasets without data leakage, ensuring reliable evaluation?
• RQ5 How does LLM-patched evaluation compare to baseline methods that treat unjudged documents as non-relevant?

Key findings

• When only 10% of relevance judgments are retained, Vicuña-7B achieves a Kendall τ of 0.862 and GPT-3.5 Turbo achieves 0.927 in correlation with ground-truth judgments on TREC DL 2019.
• The LLM-based method significantly outperforms the baseline of labeling all holes as non-relevant (τ = 0.624), achieving 25–30% higher correlation on average.
• Few-shot prompting yields more stable and higher correlation than zero-shot prompting, with lower variance across trials.
• The framework maintains strong performance on unseen data, as shown by BM25 results evaluated with LLM-patched qrels on TREC DL 2023, achieving nDCG@10 of 0.2721 with GPT-3.5 Turbo—close to the ground-truth of 0.2627.
• The method shows strong generalization, with Kendall τ values above 0.87 across all three TREC DL tracks, even under extreme hole conditions (90% missing).
• Data contamination tests confirm that LLMs are not simply memorizing the test data, as performance remains strong on TREC DL 2023, which was not used in prompting or fine-tuning.