[Paper Review] RewardBench: Evaluating Reward Models for Language Modeling

Nathan Lambert, Valentina Pyatkin+10 more|arXiv (Cornell University)|Mar 20, 2024

Topic Modeling4 citations

TL;DR

RewardBench introduces a comprehensive benchmark dataset and codebase to evaluate reward models (RMs) used in reinforcement learning from human feedback (RLHF) for language modeling. It evaluates diverse RMs—including classifier-based and Direct Preference Optimization (DPO)-trained models—across structured, out-of-distribution prompts in safety, reasoning, and chat domains, revealing key limitations in refusal behavior, reasoning, and generalization, especially for DPO models on standard test sets.

ABSTRACT

Reward models (RMs) are at the crux of successfully using RLHF to align pretrained models to human preferences, yet there has been relatively little study that focuses on evaluation of those models. Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models and which values are embedded in them. Resources for reward model training and understanding are sparse in the nascent open-source community around them. To enhance scientific understanding of reward models, we present RewardBench, a benchmark dataset and code-base for evaluation. The RewardBench dataset is a collection of prompt-chosen-rejected trios spanning chat, reasoning, and safety, to benchmark how reward models perform on challenging, structured and out-of-distribution queries. We create specific comparison datasets for RMs that have subtle, but verifiable reasons (e.g. bugs, incorrect facts) why one answer should be preferred to another. On the RewardBench leaderboard, we evaluate reward models trained with a variety of methods, such as the direct MLE training of classifiers and the implicit reward modeling of Direct Preference Optimization (DPO). We present many findings on propensity for refusals, reasoning limitations, and instruction following shortcomings of various reward models towards a better understanding of the RLHF process.

Motivation & Objective

• To address the lack of systematic evaluation for reward models (RMs), which are central to RLHF but under-studied despite their role in aligning LLMs with human preferences.
• To create a unified benchmark dataset and codebase for evaluating diverse RM architectures, including both classifier-based and DPO-trained models, across structured, out-of-distribution queries.
• To map the current landscape of open-weight reward models by evaluating their performance on a standardized leaderboard with diverse, human-verified prompt-chosen/rejected trios.
• To investigate the generalization and robustness of RMs, particularly DPO-based models, on preference evaluation sets with subtle, verifiable differences between completions.
• To release a reproducible evaluation framework with full data, inference tools, and visualization support to enable deeper analysis of RM behavior and alignment properties.

Proposed method

• The authors curate a benchmark dataset of prompt-win-lose trios across three domains: chat, reasoning, and safety, each with human-verified preferred and rejected completions.
• Each data point is structured as a prompt paired with a chosen (preferred) and rejected (less preferred) completion, with intentional design to test subtle differences such as factual errors, logical flaws, or ethical concerns.
• The benchmark includes both easy samples (near 100% accuracy for state-of-the-art RMs) and hard samples (where even top models achieve only 60–70% accuracy), enabling evaluation of model robustness.
• The evaluation framework supports inference across a wide range of RMs, including those trained via direct MLE classification (e.g., UltraRM, Starling) and DPO-fine-tuned models (e.g., Zephyr, Qwen-Chat, Tulu 2).
• A public leaderboard is maintained at HF.co/spaces/allenai/reward-bench, enabling ongoing model comparison and tracking of performance across datasets.
• The codebase provides a common inference stack, visualization tools, and access to all text-score pairs for further analysis of RM behavior and decision boundaries.

Figure 1: The scoring method of the RewardBench evaluation suite. Each prompt is accompanied by a chosen and rejected completion which are independently rated by a reward model.

Experimental results

Research questions

• RQ1 How do classifier-based reward models compare to DPO-trained models in terms of generalization and robustness across diverse, out-of-distribution preference evaluation tasks?
• RQ2 To what extent do reward models exhibit refusal behavior, and how does this vary across different types of prompts (e.g., sensitive topics, hypothetical violence)?
• RQ3 What are the limitations of current reward models in reasoning and factual consistency, especially when faced with subtle logical or factual errors in completions?
• RQ4 How well do DPO-based models perform on standard preference evaluation benchmarks, and do they generalize beyond their fine-tuning distribution?
• RQ5 What are the key failure modes of reward models, and how do they correlate with specific prompt categories such as safety, reasoning, or instruction following?

Key findings

• DPO-trained models exhibit higher variance in performance and fail to generalize effectively to standard preference evaluation test sets, despite their popularity and ease of training.
• Classifier-based reward models, such as UltraRM and Starling, consistently outperform DPO models on the same benchmarks, particularly on hard, out-of-distribution samples.
• Many reward models show high refusal rates on prompts involving violence or sensitive topics—even when the prompt is fictional or hypothetical—indicating overzealous safety alignment.
• On the XSTest Should Respond subset, models frequently refuse to answer even when the prompt is non-harmful or fictional, revealing a propensity for over-refusal in ambiguous or edge-case scenarios.
• State-of-the-art models achieve only 60–70% accuracy on the most challenging samples, indicating that current RMs still struggle with subtle reasoning and factual consistency.
• The benchmark reveals that even high-performing RMs fail on prompts with minor factual errors or logical inconsistencies, suggesting that reward models may not reliably capture nuanced human preferences.

Figure 2: The distribution of rewards outputted by reward models for the chosen and rejected responses in the RewardBench dataset. A large variety of model behaviors exist among open reward models. Some top scoring models, such as Starling and UltraRM show an increased margin between the mean of the