[Paper Review] Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators

Yinhong Liu, Han Zhou+5 more|arXiv (Cornell University)|Mar 25, 2024

Interpreting and Communication in Healthcare6 citations

TL;DR

This paper reframes LLM-based evaluation as a pairwise preference ranking problem and proposes PairS, an uncertainty-guided search method that efficiently ranks candidate texts to better align with human judgments.

ABSTRACT

Large Language Models (LLMs) have demonstrated promising capabilities as automatic evaluators in assessing the quality of generated natural language. However, LLMs still exhibit biases in evaluation and often struggle to generate coherent evaluations that align with human assessments. In this work, we first conduct a systematic study of the misalignment between LLM evaluators and human evaluation, revealing that existing calibration methods aimed at mitigating biases of LLMs are insufficient for effectively aligning LLM evaluators. Inspired by the use of preference data in RLHF, we formulate the evaluation as a ranking problem and introduce Pairwise-preference Search (PAIRS), an uncertainty-guided search-based rank aggregation method that employs LLMs to conduct pairwise comparisons locally and efficiently ranks candidate texts globally. PAIRS achieves state-of-the-art performance on representative evaluation tasks in long-form generations and demonstrates significant improvements over direct scoring. Furthermore, we provide insights into the role of pairwise preference in quantifying the transitivity of LLMs and demonstrate how PAIRS benefits from calibration using debiased pairwise evaluations.

Motivation & Objective

• Assess why score-based LLM evaluators misalign with human judgments even after calibration.
• Propose a pairwise-preference evaluation framework inspired by RLHF to improve alignment.
• Develop PairS, an uncertainty-guided search algorithm to efficiently rank candidate texts via pairwise comparisons.
• Analyze transitivity as a property of LLM evaluators and how calibration interacts with pairwise methods.
• Demonstrate scalability and robustness of PairS across summarization and open-ended generation tasks.

Proposed method

• Systematically analyze calibration effectiveness and identify misalignment in likelihoods p(y|s) between LLMs and humans.
• Formulate evaluation as a ranking problem and define a transitive likelihood L_T for pairwise comparisons.
• Introduce PairS (Pair wise-preference Search) with PairS-greedy and PairS-beam variants to efficiently search the space of pairwise comparisons.
• Incorporate an uncertainty-based pruning mechanism using entropy to guide pruning of low-uncertainty comparisons.
• Provide a two-stage scaling variant to handle large evaluation sets with N log N to O(N) behavior after anchoring.
• Apply Batch Calibration on PairS with a uniform human prior and a marginal model prior to reduce biases.

Figure 1: Illustration of RLHF (left), direct scoring with LLM evaluators (middle), and pairwise preference search ( PairS ; right). Pairwise preference data have been utilized to train the reward model to align the LLM in RLHF. Leveraging this idea, PairS reframes the traditional scoring-based eval

Experimental results

Research questions

• RQ1 Can pairwise preference data yield evaluations that align more closely with human judgments than direct scoring for LLM evaluators?
• RQ2 How can uncertainty-guided search efficiently estimate the maximum likelihood ranking in a pairwise comparison space?
• RQ3 What is the impact of transitivity and calibration on the performance and reliability of LLM evaluators?
• RQ4 How scalable is PairS to large evaluation sets and real-world tasks like summarization and story generation?

Key findings

• Calibration of score-based LLM evaluators does not fully align with human judgments; misalignment persists after calibration.
• PairS substantially improves alignment with human judgments across summarization and story generation tasks, often approaching or matching state-of-the-art baselines in a zero-shot setting.
• PairS-beam generally yields more robust rankings (higher correlations, lower standard errors) than PairS-greedy, particularly for models with lower transitivity.
• Uncertainty-guided pruning reduces search space and maintains ranking quality, enabling scalable evaluation.
• Transitivity of LLM evaluators correlates with evaluation quality; stronger transitivity leads to more efficient and reliable rankings, with GPT-3.5 showing notably consistent behavior in experiments.
• Calibration helps PairS, but its gains vary by model (more benefit for smaller models than for highly capable ones).

Figure 2: LLM evaluations are misaligned with human judgements. The score histograms on evaluating the coherence in HANNA (Chhun et al., 2022 ) and SummEval (Fabbri et al., 2021 ) . We present the scores from ground human evaluations, LLMs, and LLMs after calibrations. The histograms can be interpre