[Paper Review] Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge

Jiayi Ye, Yanbo Wang+10 more|arXiv (Cornell University)|Oct 3, 2024

Law, Economics, and Judicial Systems5 citations

TL;DR

This paper defines 12 biases in LLMs used as judges, introduces Calm, an automated framework to quantify and analyze these biases, and evaluates six LLMs to reveal persistent biases and reliability limitations.

ABSTRACT

LLM-as-a-Judge has been widely utilized as an evaluation method in various benchmarks and served as supervised rewards in model training. However, despite their excellence in many domains, potential issues are under-explored, undermining their reliability and the scope of their utility. Therefore, we identify 12 key potential biases and propose a new automated bias quantification framework-CALM-which systematically quantifies and analyzes each type of bias in LLM-as-a-Judge by using automated and principle-guided modification. Our experiments cover multiple popular language models, and the results indicate that while advanced models have achieved commendable overall performance, significant biases persist in certain specific tasks. Empirical results suggest that there remains room for improvement in the reliability of LLM-as-a-Judge. Moreover, we also discuss the explicit and implicit influence of these biases and give some suggestions for the reliable application of LLM-as-a-Judge. Our work highlights the need for stakeholders to address these issues and remind users to exercise caution in LLM-as-a-Judge applications.

Motivation & Objective

• Define and categorize 12 biases that can affect LLMs acting as judges.
• Propose Calm, an automated, perturbation-based framework to quantify judge biases.
• Evaluate multiple LLMs to assess robustness and reliability of judgments under bias.
• Provide guidance for reliable deployment of LLM-as-a-Judge in benchmarks and rewards.

Proposed method

• Introduce Calm (Comprehensive Assessment of Language Model Judge Biases) with four components: bias taxonomy, diverse evaluation datasets, bias-specific metrics, and automated perturbations.
• Use an attack-and-detect approach where biases are injected via principle-guided perturbations g(·) to R or I and judged for consistency.
• Apply both scoring and pairwise comparison judging tasks across fact-related, refinement-aware, and alignment datasets.
• Define metrics such as Robustness Rate (RR), Consistency Rate (CR), and accuracy measures to quantify bias impact.
• Evaluate six LLMs (ChatGPT, GPT-4-Turbo, GPT-4o, Claude-3.5, GLM-4, Qwen2) under multiple bias scenarios.

Experimental results

Research questions

• RQ1 What are the 12 distinct biases that can affect LLMs used as judges?
• RQ2 How can automated perturbations quantify the robustness and reliability of LLM-based judgments?
• RQ3 Do modern LLMs exhibit persistent biases across different judging tasks and datasets?
• RQ4 What guidance and mitigation strategies can improve the reliability of LLM-as-a-Judge in practice?

Key findings

• Biases significantly affect judging robustness; even strong models show vulnerabilities that are task- and dataset-dependent.
• Alignment data show stronger bias effects than fact-related data, indicating dataset quality influences judge reliability.
• Claude-3.5 generally shows higher resilience to biases, but no model is universally robust across all bias types.
• Position, verbosity, and self-enhancement biases are prominent, with self-enhancement showing clear divergence between generation and evaluation sources.
• CoT (chain-of-thought) can improve evaluation accuracy for some models but not universally.