[论文解读] Who can we trust? LLM-as-a-jury for Comparative Assessment

Mengjie Qian, Guangzhi Sun+2 more|arXiv (Cornell University)|Feb 18, 2026

Topic Modeling被引用 0

一句话总结

引入 BT-σ，一种考虑评判者的 Bradley–Terry 模型扩展，用于仅从成对LLM比较中联合推断项排序与评判者可靠性，相较简单平均和标定的聚合提高了鲁棒性

ABSTRACT

Large language models (LLMs) are increasingly applied as automatic evaluators for natural language generation assessment often using pairwise comparative judgements. Existing approaches typically rely on single judges or aggregate multiple judges assuming equal reliability. In practice, LLM judges vary substantially in performance across tasks and aspects, and their judgment probabilities may be biased and inconsistent. Furthermore, human-labelled supervision for judge calibration may be unavailable. We first empirically demonstrate that inconsistencies in LLM comparison probabilities exist and show that it limits the effectiveness of direct probability-based ranking. To address this, we study the LLM-as-a-jury setting and propose BT-sigma, a judge-aware extension of the Bradley-Terry model that introduces a discriminator parameter for each judge to jointly infer item rankings and judge reliability from pairwise comparisons alone. Experiments on benchmark NLG evaluation datasets show that BT-sigma consistently outperforms averaging-based aggregation methods, and that the learned discriminator strongly correlates with independent measures of the cycle consistency of LLM judgments. Further analysis reveals that BT-sigma can be interpreted as an unsupervised calibration mechanism that improves aggregation by modelling judge reliability.

研究动机与目标

• Motivate the unreliability and bias in single-LLM or uniformly-weighted judge aggregation for comparative NLG evaluation.
• Propose a probabilistic, judge-aware ranking model that learns item rankings and judge reliability from pairwise comparisons alone.
• Demonstrate that BT-σ outperforms averaging-based methods and supervised calibration on benchmark NLG datasets.
• Analyze learned judge discriminators as unsupervised indicators of judge reliability and consistency.

提出的方法

• Model pairwise comparisons with a soft Bradley–Terry framework to obtain a global item ranking.
• Introduce judge-specific discriminators σ_k so that P_k(i≻j)=σ((s_i−s_j)/σ_k), learning s_i and σ_k jointly without labels.
• Apply a debiasing step to enforce commutativity p'_{ij} = 0.5(p_{ij}+(1−p_{ji})).
• Optionally extend to aspect-dependent discriminators σ_k,asp to allow reliability to vary by evaluation aspect.
• Compare against Avg-Prob, hard BT, soft BT, and Temp-BT across SummEval and Topical-Chat using Spearman correlations with human judgments.
• Demonstrate that BT-σ upweights reliable judges and downweights noisy signals, improving robustness.

实验结果

研究问题

• RQ1 How do inconsistencies in LLM pairwise probabilities affect ranking quality when aggregating judgments?
• RQ2 Can a judge-aware BT model jointly infer item rankings and judge reliability from pairwise comparisons without human labels?
• RQ3 Does BT-σ consistently outperform averaging and calibration baselines on reference-free NLG evaluation benchmarks?
• RQ4 Do learned judge discriminators correlate with independent measures of judge reliability and cycle consistency?

主要发现

• BT-based aggregation improves ranking stability over direct averaging across most models and aspects.
• BT-σ consistently outperforms soft BT and hard BT in aggregated evaluations on SummEval and Topical-Chat.
• Learned judge discriminators 1/σ_k correlate positively with judge performance and with lower cycle inconsistency, indicating effective unsupervised reliability signals.
• BT-σ provides robust performance when probability signals are noisy and inconsistent, by downweighting unreliable judges.
• BT-σ-asp offers marginal per-aspect gains, suggesting a single judge discriminator suffices in practice.
• Hard BT-σ can outperform soft variants in highly inconsistent settings (e.g., ENG on Topical-Chat), while soft BT-σ excels under moderate inconsistency (SummEval).