[Paper Review] Self-Preference Bias in LLM-as-a-Judge

Koki Wataoka, Tsubasa Takahashi+1 more|arXiv (Cornell University)|Oct 29, 2024

Law, Economics, and Judicial SystemsEconomics, Econometrics and Finance3 citations

TL;DR

This paper introduces a novel fairness-based metric to quantitatively measure self-preference bias in LLM-as-a-judge systems, demonstrating that GPT-4 exhibits significant bias by favoring its own outputs. The study links this bias to lower perplexity in generated texts, revealing that LLMs prefer responses more familiar to them, regardless of whether they originated from the model itself.

ABSTRACT

Automated evaluation leveraging large language models (LLMs), commonly referred to as LLM evaluators or LLM-as-a-judge, has been widely used in measuring the performance of dialogue systems. However, the self-preference bias in LLMs has posed significant risks, including promoting specific styles or policies intrinsic to the LLMs. Despite the importance of this issue, there is a lack of established methods to measure the self-preference bias quantitatively, and its underlying causes are poorly understood. In this paper, we introduce a novel quantitative metric to measure the self-preference bias. Our experimental results demonstrate that GPT-4 exhibits a significant degree of self-preference bias. To explore the causes, we hypothesize that LLMs may favor outputs that are more familiar to them, as indicated by lower perplexity. We analyze the relationship between LLM evaluations and the perplexities of outputs. Our findings reveal that LLMs assign significantly higher evaluations to outputs with lower perplexity than human evaluators, regardless of whether the outputs were self-generated. This suggests that the essence of the bias lies in perplexity and that the self-preference bias exists because LLMs prefer texts more familiar to them.

Motivation & Objective

• To address the lack of reliable, quantitative metrics for measuring self-preference bias in LLM-as-a-judge systems.
• To investigate the underlying causes of self-preference bias, particularly whether familiarity—measured by perplexity—drives the bias.
• To compare LLM evaluations against human judgments in pairwise settings to isolate bias patterns.
• To develop a metric grounded in algorithmic fairness concepts for consistent, interpretable bias measurement.
• To evaluate the extent of self-preference bias across eight diverse LLMs, with a focus on GPT-4’s behavior.

Proposed method

• Proposes a new metric for self-preference bias based on the fairness concept of Equal Opportunity, defining bias as the difference in evaluation outcomes when a model judges its own output versus a non-self output.
• Employs a pairwise evaluation framework where each LLM judges two responses: one generated by itself and one by another model, under identical prompts.
• Measures the winning rate of self-generated responses across multiple prompts to compute the bias score using the proposed fairness-based definition.
• Analyzes the correlation between perplexity of evaluated texts and the scores assigned by LLMs, comparing these to human evaluator preferences.
• Applies an alternative fairness metric—Demographic Parity—to validate consistency and cross-validate findings.
• Uses statistical analysis to compare LLM evaluations with human judgments, isolating the effect of perplexity on evaluation scores.

(a)

Experimental results

Research questions

• RQ1 To what extent do LLMs exhibit self-preference bias when evaluating their own outputs compared to others in a pairwise setting?
• RQ2 Is the self-preference bias in LLM-as-a-judge systems correlated with the perplexity of the evaluated text?
• RQ3 Do LLMs assign significantly higher scores to low-perplexity texts than human evaluators, regardless of whether the text was self-generated?
• RQ4 Can the self-preference bias be quantitatively measured using fairness-based metrics, and how do different LLMs compare in this regard?
• RQ5 What underlying mechanism—such as familiarity or alignment with internal policies—drives the observed bias in high-performing models like GPT-4?

Key findings

• GPT-4 exhibits a significant self-preference bias, with a self-preference bias score of 0.749 using the Demographic Parity metric, indicating it rates its own outputs much more favorably than others.
• The proposed fairness-based metric (Definition 4.1) successfully quantifies self-preference bias, revealing that GPT-4’s bias is substantially higher than that of other models like GPT-3.5-turbo (0.191) and Vicuna-13b (0.382).
• LLMs assign significantly higher evaluations to texts with lower perplexity than human evaluators do, even when the text was not self-generated, indicating that familiarity—measured by perplexity—drives the bias.
• The bias is not solely due to self-generation but is rooted in the model’s familiarity with low-perplexity sequences, suggesting that perceptual similarity to training data influences evaluation outcomes.
• The study confirms that self-preference bias is not limited to factual errors but is often tied to stylistic differences, such as response structure or policy adherence, especially in advanced models like GPT-4.
• The results suggest that ensemble evaluation using multiple LLMs, with reduced weighting for models that assign high scores to low-perplexity outputs, could mitigate self-preference bias in practice.

(b)