[Paper Review] Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges

Aman Singh Thakur, Kartik Choudhary+3 more|arXiv (Cornell University)|Jun 18, 2024

Law, Economics, and Judicial Systems6 citations

TL;DR

This paper systematically studies LLMs acting as judges for TriviaQA, comparing them to human judgments and lexical baselines; it finds that only a few judges align well with humans, Cohen’s kappa is a better alignment metric than percent agreement, and some cheaper methods can rival or outperform in ranking exam-taker models.

ABSTRACT

Offering a promising solution to the scalability challenges associated with human evaluation, the LLM-as-a-judge paradigm is rapidly gaining traction as an approach to evaluating large language models (LLMs). However, there are still many open questions about the strengths and weaknesses of this paradigm, and what potential biases it may hold. In this paper, we present a comprehensive study of the performance of various LLMs acting as judges, focusing on a clean scenario in which inter-human agreement is high. Investigating thirteen judge models of different model sizes and families, judging answers of nine different 'examtaker models' - both base and instruction-tuned - we find that only the best (and largest) models achieve reasonable alignment with humans. However, they are still quite far behind inter-human agreement and their assigned scores may still differ with up to 5 points from human-assigned scores. In terms of their ranking of the nine exam-taker models, instead, also smaller models and even the lexical metric contains may provide a reasonable signal. Through error analysis and other studies, we identify vulnerabilities in judge models, such as their sensitivity to prompt complexity and length, and a tendency toward leniency. The fact that even the best judges differ from humans in this comparatively simple setup suggest that caution may be wise when using judges in more complex setups. Lastly, our research rediscovers the importance of using alignment metrics beyond simple percent alignment, showing that judges with high percent agreement can still assign vastly different scores.

Motivation & Objective

• Assess the alignment between LLM judges and human judgments on a knowledge benchmark (TriviaQA).
• Compare multiple judge models across sizes and families, including base and instruction-tuned variants.
• Identify biases, errors, and reliability issues in LLM-based judging.
• Evaluate whether cheaper or specialized judge models can match or outperform larger judges in ranking exam-taker models.

Proposed method

• Use TriviaQA validation set with 400 questions for manual exam-taker annotations; 9 judge models evaluate 9 exam-taker models.
• Compare judge outputs to human judgments using percent agreement and Cohen’s kappa.
• Baseline lexical metrics include exact match (EM) and contains match (contains).
• Examine rank correlations with human judgments using Spearman’s rho across nine exam-taker models.
• Analyze error types, prompt variation effects, and robustness to dummy or hallucinated responses.

(a)

Experimental results

Research questions

• RQ1 How well do different LLMs function as judges align with human judgments on TriviaQA questions?
• RQ2 Does Cohen’s kappa provide a more reliable alignment measure than percent agreement for LLM judges?
• RQ3 Which judge models best differentiate among exam-taker models in terms of ranking, and at what cost in alignment?
• RQ4 What systematic biases or failure modes affect LLM judges (e.g., prompt length, leniency, or sensitivity to non-informative answers)?

Key findings

• GPT-4 Turbo and Llama-3 70B show excellent alignment with humans (kappa 84 and 79 respectively), but remain below human alignment levels (96).
• Contains achieves higher kappa than EM for several judges, yet EM often performs worst overall; percent agreement can be misleading.
• Contains, JudgeLM 7B, and GPT-4 Turbo/Llama-3 70B differ in ranking accuracy; some cheaper methods can match or exceed in ranking accuracy despite lower alignment.
• Judge models generally struggle with under-specified or overly lenient judgments and can be fooled by non-informative answers like “Yes” or “Sure.”
• Recall improves with higher human alignment (r^2 ≈ 0.98) while precision shows no clear trend with alignment.
• Prompts with more detailed guidelines help only the top-performing judges; many judges are sensitive to prompt design and reference order.

(b)