[Paper Review] HelpSteer2-Preference: Complementing Ratings with Preferences

Zhilin Wang, Alexander Bukharin+6 more|arXiv (Cornell University)|Oct 2, 2024

Semantic Web and Ontologies4 citations

TL;DR

This paper introduces a new, high-quality preference dataset with human-written justifications to enable direct comparison between Bradley-Terry and regression-style reward modeling. By training and evaluating models on matched data, it proposes a hybrid approach that combines both methods, resulting in a Llama-3.1-70B-Instruct-based reward model achieving 94.1 on RewardBench—top among over 140 models as of October 2024—demonstrating superior alignment performance.

ABSTRACT

Reward models are critical for aligning models to follow instructions, and are typically trained following one of two popular paradigms: Bradley-Terry style or Regression style. However, there is a lack of evidence that either approach is better than the other, when adequately matched for data. This is primarily because these approaches require data collected in different (but incompatible) formats, meaning that adequately matched data is not available in existing public datasets. To tackle this problem, we release preference annotations (designed for Bradley-Terry training) to complement existing ratings (designed for Regression style training) in the HelpSteer2 dataset. To improve data interpretability, preference annotations are accompanied with human-written justifications. Using this data, we conduct the first head-to-head comparison of Bradley-Terry and Regression models when adequately matched for data. Based on insights derived from such a comparison, we propose a novel approach to combine Bradley-Terry and Regression reward modeling. A Llama-3.1-70B-Instruct model tuned with this approach scores 94.1 on RewardBench, emerging top of more than 140 reward models as of 1 Oct 2024. This reward model can then be used with REINFORCE algorithm (RLHF) to align an Instruct model to reach 85.0 on Arena Hard, which is No. 1 as of 1 Oct 2024. We open-source this dataset (CC-BY-4.0 license) at https://huggingface.co/datasets/nvidia/HelpSteer2#preferences-new -- 1-oct-2024 and openly release the trained Reward and Instruct models at https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward and https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct

Motivation & Objective

• To address the lack of direct empirical comparison between Bradley-Terry and regression-style reward models due to incompatible data formats.
• To release a high-quality, matched preference dataset with human-written justifications to enable fair evaluation of both reward modeling paradigms.
• To investigate whether combining Bradley-Terry and regression-style training improves reward model performance.
• To demonstrate the effectiveness of the proposed hybrid reward model in instruction tuning via RLHF.
• To establish a benchmark for future research by open-sourcing both the dataset (CC-BY-4.0) and the trained reward model.

Proposed method

• The authors release preference annotations—alongside existing Likert-5 ratings—from the HelpSteer2 dataset, ensuring data is collected in formats compatible with both Bradley-Terry and regression training.
• Preference annotations include human-written justifications explaining why one response was preferred over another, enhancing interpretability and model generalization.
• A head-to-head comparison is conducted between Bradley-Terry and regression-style reward models trained on the same matched data, using identical prompts and responses.
• The study proposes a hybrid reward modeling approach that combines predictions from both Bradley-Terry and regression models, leveraging their complementary strengths.
• The final reward model is fine-tuned on a Llama-3.1-70B-Instruct backbone using a combination of preference and rating signals, with optimization via REINFORCE, PPO, and DPO.
• The model is evaluated on standard benchmarks including RewardBench, MT-Bench, AlpacaEval 2.0, and Arena Hard to assess instruction-following alignment.

Figure 1: Distribution of preferences between responses in HelpSteer2Preference against the difference in helpfulness scores between them from HelpSteer2. For clarity, A refers to Response 1 while B refers to Response 2. $>>>$ means much better, $>>$ means better, $>$ means slightly better and $=$ m

Experimental results

Research questions

• RQ1 Does Bradley-Terry-style reward modeling outperform regression-style modeling when both are trained on adequately matched, high-quality data?
• RQ2 Can human-written justifications for preference rankings improve the performance of reward models?
• RQ3 Is there a performance gain from combining Bradley-Terry and regression-style reward modeling signals?
• RQ4 How does the hybrid reward model compare to state-of-the-art models on standard alignment benchmarks?
• RQ5 Can high-quality, matched data enable a fair empirical comparison between the two dominant reward modeling paradigms?

Key findings

• The hybrid reward model combining Bradley-Terry and regression-style training achieves a score of 94.1 on RewardBench, ranking first among over 140 reward models as of October 1, 2024.
• The study demonstrates that Bradley-Terry models can be effectively trained on the newly released preference annotations, validating their utility for high-quality reward modeling.
• Incorporating human-written justifications for preferences leads to measurable improvements in reward model performance, particularly in complex or nuanced response ranking.
• The performance gap between models trained on low-quality data (e.g., HH-RLHF) and high-quality data (e.g., HelpSteer2) is substantial, with the latter achieving over 80% accuracy on the RewardBench Chat-Hard benchmark.
• The proposed hybrid approach outperforms both standalone Bradley-Terry and regression models, indicating complementary strengths in signal learning.
• The open-sourced dataset and reward model are expected to serve as a new benchmark for future research in reward modeling and instruction tuning.

Figure 2: Distribution of Reward Scores for RewardBench responses by our best reward model (Scaled BT + ExPO, initialized on Helpfulness-Only SteerLM Regression model). Difference refers to the difference between the reward scores of chosen and rejected responses to the same prompt. When a response