[Paper Review] Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge

Kayla Schroeder, Zach Wood-Doughty|arXiv (Cornell University)|Dec 17, 2024

Legal Systems and Judicial Processes4 citations

TL;DR

This paper introduces a rigorous statistical framework using McDonald’s omega to evaluate the internal consistency reliability of LLM-as-a-judge systems. It demonstrates that even with fixed temperature settings, single-shot LLM judgments exhibit significant variability, and reliability metrics are essential for trustworthy evaluation in high-stakes applications like AI safety and content moderation.

ABSTRACT

Large Language Models (LLMs) have become increasingly powerful and ubiquitous, but their stochastic nature poses challenges to the reliability of their outputs. While deterministic settings can improve consistency, they do not guarantee reliability, as a single sample from the model's probability distribution can still be misleading. Building upon the concept of LLM-as-a-judge, we introduce a novel framework for rigorously evaluating the reliability of LLM judgments, leveraging McDonald's omega. We evaluate the reliability of LLMs when judging the outputs of other LLMs on standard single-turn and multi-turn benchmarks, simultaneously investigating the impact of temperature on reliability. By analyzing these results, we demonstrate the limitations of fixed randomness and the importance of considering multiple samples, which we show has significant implications for downstream applications. Our findings highlight the need for a nuanced understanding of LLM reliability and the potential risks associated with over-reliance on single-shot evaluations. This work provides a crucial step towards building more trustworthy and reliable LLM-based systems and applications.

Motivation & Objective

• To address the critical gap in assessing the reliability of LLM-as-a-judge systems, especially in high-stakes applications like content moderation.
• To challenge the assumption that deterministic settings (fixed temperature) ensure reliable LLM judgments.
• To introduce McDonald’s omega as a robust, statistically grounded metric for quantifying internal consistency reliability in LLM judgments.
• To demonstrate that high reliability values do not eliminate variability across replications, especially in complex, multi-turn tasks.
• To advocate for reporting reliability metrics alongside LLM-as-a-judge results to improve transparency and trust in AI evaluation.

Proposed method

• Adopt McDonald’s omega as the primary reliability metric to quantify internal consistency across multiple LLM judgments on the same input.
• Apply the framework to standard single-turn and multi-turn benchmarks, including BBH, SQuAD, MT-Bench, and Head-to-Tail.
• Evaluate LLM judgments under varying temperature settings to assess sensitivity to stochasticity, even in deterministic configurations.
• Collect multiple judgments per input to compute reliability scores, simulating repeated evaluations to capture variability.
• Use statistical inference to report uncertainty in LLM judgments, analogous to confidence intervals in traditional research.
• Compare reliability scores across different benchmarks and task types to identify patterns in judgment instability.

Experimental results

Research questions

• RQ1 To what extent do fixed-temperature LLMs produce reliable judgments across multiple runs, despite deterministic settings?
• RQ2 How does temperature influence the reliability of LLM-as-a-judge outputs across different benchmark types?
• RQ3 Can McDonald’s omega effectively quantify the internal consistency reliability of LLM judgments in subjective and complex evaluation tasks?
• RQ4 How do reliability scores correlate with observed variability in downstream LLM evaluation results?
• RQ5 What are the implications of low reliability for real-world applications such as AI safety and content moderation?

Key findings

• Even with fixed temperature settings, LLM-as-a-judge outputs exhibit significant variability across replications, indicating that deterministic configurations do not guarantee reliability.
• Reliability scores computed via McDonald’s omega reveal substantial differences across benchmarks, with multi-turn and complex tasks showing notably lower reliability.
• High reliability values do not eliminate variability in downstream outputs, as demonstrated in Figure 3, where high-reliability models still produce inconsistent judgments across runs.
• The study finds that single-shot evaluations are insufficient for trust in LLM judgments, especially in subjective or high-stakes domains.
• Reliability metrics should be reported alongside LLM-as-a-judge results to inform users of the uncertainty inherent in model outputs.
• The framework demonstrates that reliability is a critical, quantifiable dimension of LLM evaluation that must be integrated into research and deployment practices.