[Paper Review] An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4

Hui Huang, Bu, Xingyuan+6 more|arXiv (Cornell University)|Mar 5, 2024

Artificial Intelligence in LawSocial Sciences11 citations

TL;DR

The paper empirically evaluates fine-tuned judge models for LLM evaluation and shows they excel in-domain but are task-specific, overfit to schemes, and underperform GPT-4 in generalizability and fairness.

ABSTRACT

Recently, there has been a growing trend of utilizing Large Language Model (LLM) to evaluate the quality of other LLMs. Many studies have fine-tuned judge models based on open-source LLMs for evaluation. While the fine-tuned judge models are claimed to achieve comparable evaluation capability with GPT-4, in this work, we conduct an empirical study of LLM-as-a-Judge. Our findings indicate that although the fine-tuned judge models achieve high performance on in-domain test sets, even surpassing GPT-4, they underperform GPT-4 across several dimensions, including generalizability, fairness and adaptability. We also reveal that the fine-tuned judge model inherently operates as a task-specific classifier, consequently imposing the limitations.

Motivation & Objective

• Assess the evaluation capability of fine-tuned judge models across multiple datasets and schemes.
• Compare fine-tuned judges to GPT-4 in accuracy, agreement, and fairness.
• Investigate whether generation-style versus classification-style judges impacts performance.
• Examine generalizability and bias of fine-tuned judges under cross-scheme evaluation.

Proposed method

• Fine-tune four open-source judge models (JudgeLM, PandaLM, Auto-J, Prometheus) on data from GPT-4 or human annotations.
• Train both generation-style and classification-style (regression) variants using the same data and prompts.
• Evaluate on in-domain and cross-domain testsets, including JudgeLM-test, PandaLM-test, Auto-J-test, Prometheus-test, MT-bench, and LLMBar bias tests.
• Compare against GPT-4 and other baselines using accuracy, F1, agreement, and Pearson metrics.
• Analyze correlations across evaluators and across schemes to assess overfitting and generalizability.

Experimental results

Research questions

• RQ1 Do fine-tuned judge models achieve comparable accuracy to GPT-4 on in-domain evaluation sets?
• RQ2 Are fine-tuned judges inherently task-specific classifiers with limited generalizability across schemes?
• RQ3 Do generation-style and classification-style judges differ in performance or biases?
• RQ4 How do fine-tuned judges perform on cross-scheme and bias evaluation datasets compared to GPT-4?
• RQ5 What biases (e.g., towards verbosity or superficial quality) affect fine-tuned judges, and how do they compare to GPT-4?

Key findings

• Fine-tuned judges achieve high accuracy on in-domain test sets but are limited by scheme-specific overfitting.
• Classification-style judges perform comparably to generation-style ones when trained on the same data.
• Judges trained on one evaluation scheme perform poorly when applied to a different scheme, unlike GPT-4 which remains robust.
• On cross-domain and multi-turn MT-bench evaluation, fine-tuned judges underperform GPT-4 by a large margin.
• Fine-tuned judges exhibit biases toward superficial qualities such as formality or verbosity, unlike GPT-4.
• DeBERTa-based evaluator shows better fairness than LLM-based evaluators in bias tests.