[Paper Review] RLHF Workflow: From Reward Modeling to Online RLHF

Hanze Dong, Wei Xiong+8 more|arXiv (Cornell University)|May 13, 2024

Simulation Techniques and ApplicationsDecision Sciences3 citations

TL;DR

This paper presents a reproducible, fully open-source workflow for online iterative Reinforcement Learning from Human Feedback (RLHF), using a proxy preference model trained on diverse open datasets to approximate human feedback. The method achieves state-of-the-art performance on benchmarks like AlpacaEval-2, Arena-Hard, and MT-Bench, with the SFR-Iterative-DPO-LLaMA-3-8B-R model outperforming offline baselines through iterative on-policy optimization and length penalty mitigation.

ABSTRACT

We present the workflow of Online Iterative Reinforcement Learning from Human Feedback (RLHF) in this technical report, which is widely reported to outperform its offline counterpart by a large margin in the recent large language model (LLM) literature. However, existing open-source RLHF projects are still largely confined to the offline learning setting. In this technical report, we aim to fill in this gap and provide a detailed recipe that is easy to reproduce for online iterative RLHF. In particular, since online human feedback is usually infeasible for open-source communities with limited resources, we start by constructing preference models using a diverse set of open-source datasets and use the constructed proxy preference model to approximate human feedback. Then, we discuss the theoretical insights and algorithmic principles behind online iterative RLHF, followed by a detailed practical implementation. Our trained LLM achieves impressive performance on LLM chatbot benchmarks, including AlpacaEval-2, Arena-Hard, and MT-Bench, as well as other academic benchmarks such as HumanEval and TruthfulQA. We have shown that supervised fine-tuning (SFT) and iterative RLHF can obtain state-of-the-art performance with fully open-source datasets. Further, we have made our models, curated datasets, and comprehensive step-by-step code guidebooks publicly available. Please refer to https://github.com/RLHFlow/RLHF-Reward-Modeling and https://github.com/RLHFlow/Online-RLHF for more detailed information.

Motivation & Objective

• To close the gap between closed-source and open-source RLHF by enabling online, iterative RLHF in resource-constrained settings.
• To develop a practical, reproducible pipeline for online iterative RLHF using open-source datasets as proxies for human feedback.
• To mitigate length bias in LLM responses through a learned length penalty during reward modeling.
• To demonstrate state-of-the-art performance on standard LLM benchmarks using fully open-source data and models.
• To provide comprehensive code, datasets, and hyperparameter guides to lower the barrier to entry for open-source RLHF.

Proposed method

• Construct a proxy preference model using a diverse set of open-source preference datasets to simulate human feedback.
• Apply iterative Direct Preference Optimization (DPO) with on-policy sampling to refine the policy in a closed-loop manner.
• Use a KL-regularized objective to optimize the policy, balancing reward maximization and divergence from the initial policy.
• Incorporate a length penalty in the reward function to reduce verbosity and improve response quality.
• Leverage the Bradley-Terry model to formalize preference signals as probabilistic rewards: $\mathbb{P}(a^1 \succ a^2) = \sigma(r^*(x,a^1) - r^*(x,a^2))$.
• Train the final model, SFR-Iterative-DPO-LLaMA-3-8B-R, through iterative fine-tuning with human-feedback proxies and ablation studies on reward quality and length control.

Experimental results

Research questions

• RQ1 Can online iterative RLHF be effectively implemented in open-source settings without direct human feedback?
• RQ2 How does using a proxy preference model trained on open datasets compare to human feedback in terms of alignment quality and model performance?
• RQ3 What impact does a learned length penalty have on response quality and benchmark performance in iterative RLHF?
• RQ4 How does the choice of reward model affect convergence speed, response length, and performance on academic vs. chat-based benchmarks?
• RQ5 Can iterative on-policy RLHF with proxy feedback achieve SOTA results comparable to closed-source models?

Key findings

• The SFR-Iterative-DPO-LLaMA-3-8B-R model achieves a 41.9% win rate on AlpacaEval-2, significantly outperforming the baseline without length penalty.
• The model with length penalty reduces average response length from 656 tokens to 382 tokens on the Arena-Hard benchmark, improving length control.
• The model trained with UltraRM-13B shows a higher training loss and slower convergence, indicating more complex preference signals from the proposed proxy model.
• Despite longer responses, the UltraRM-13B model achieves a higher Arena-Hard win rate (29.1 vs. 22.1), suggesting a verbosity bias in that benchmark.
• The model with length penalty achieves better results on HumanEval (+66.5) and TruthfulQA (+65.1), indicating improved reasoning and factual consistency.
• The ablation study confirms that reward model quality and length penalty are critical factors in performance, especially on reasoning and alignment benchmarks.