[Paper Review] Humans or LLMs as the Judge? A Study on Judgement Biases

Guiming Hardy Chen, Shunian Chen+3 more|arXiv (Cornell University)|Feb 16, 2024

Law, Economics, and Judicial Systems9 citations

TL;DR

The paper proposes a framework to study five judgment biases in both humans and LLMs evaluating open-ended answers, conducts extensive experiments with perturbations, and shows both groups exhibit biases that can be exploited, supported by an open-source dataset.

ABSTRACT

Adopting human and large language models (LLM) as judges (a.k.a human- and LLM-as-a-judge) for evaluating the performance of LLMs has recently gained attention. Nonetheless, this approach concurrently introduces potential biases from human and LLMs, questioning the reliability of the evaluation results. In this paper, we propose a novel framework that is free from referencing groundtruth annotations for investigating Misinformation Oversight Bias, Gender Bias, Authority Bias and Beauty Bias on LLM and human judges. We curate a dataset referring to the revised Bloom's Taxonomy and conduct thousands of evaluations. Results show that human and LLM judges are vulnerable to perturbations to various degrees, and that even the cutting-edge judges possess considerable biases. We further exploit these biases to conduct attacks on LLM judges. We hope that our work can notify the community of the bias and vulnerability of human- and LLM-as-a-judge, as well as the urgency of developing robust evaluation systems.

Motivation & Objective

• Motivate robust evaluation of LLMs by examining biases in human and LLM judges for open-ended tasks.
• Define and categorize five judge biases (Fallacy Oversight, Authority, Beauty, Verbosity, Positional) and test their impact.
• Develop an intervention/post-hoc analysis framework that does not rely on ground-truth references.
• Create and release an open dataset for open-ended evaluation to facilitate bias analysis.

Proposed method

• Design an intervention and post-hoc analysis framework to assess five biases without requiring a ground-truth standard.
• Use GPT-4 to generate questions and pairs of answers across Bloom’s revised taxonomy and collect human judgments on semantic quality.
• Perturb answers with factual errors, fake references, and rich content to measure vulnerability (Attack Successful Rate, ASR).
• Evaluate a set of human judges and representative LLMs (e.g., GPT-4, GPT-4-Turbo, Claude-2, PaLM-2, Ernie, LLaMA2, etc.) under controlled control and experimental groups.
• Compute ASR and accuracy to quantify robustness to perturbations and identify biases.
• Analyze positional and verbosity biases via post-hoc analysis and multiple evaluation rounds with shuffled answer positions.

Figure 1: Sample demonstration. Each sample consists of one question, two unperturbed answers $A_{1}$ , $A_{2}$ in the Control Group. The perturbed versions of $A_{2}$ are generated for the Experimental Group. Texts with factual errors are colored in red solely for demonstration purposes. Rich conte

Experimental results

Research questions

• RQ1 How biased are humans and LLMs when judging open-ended generation without a golden ground truth?
• RQ2 What are the manifestations and magnitudes of Fallacy Oversight, Authority, Beauty, Verbosity, and Positional biases in both humans and LLM judges?
• RQ3 How susceptible are different judges to perturbations designed to exploit these biases?
• RQ4 Can biases in LLM judges be exploited to produce superficially favorable evaluations of weaker or perturbed answers?
• RQ5 What stop-gap measures (e.g., multiple randomized position evaluations) mitigate these biases, and how can an open dataset support robust evaluation research?

Key findings

• Both human and LLM judges exhibit biases in open-ended evaluation.
• Human judges show significant Fallacy Oversight, Beauty, and Verbosity biases; LLMs exhibit varying biases across models.
• Different LLMs have distinct bias profiles, with some being more robust to certain perturbations than others.
• Bias perturbations can be used to superficially improve judgments of perturbed or weaker answers, enabling biased attacks on LLM evaluators.
• The study provides an open-source dataset of open-ended evaluation to support further bias analysis and development of robust evaluation systems.

Figure 3: Verbosity Bias of different judges. The X-Axis indicates the absolute length difference between the long answer and the short answer. Lengths are computed using tiktoken library from OpenAI. The Y-Axis indicates the preference towards the long answer. 0 refers to a total favor for the shor