[Paper Review] Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafael Rafailov, Archit Sharma+4 more|arXiv (Cornell University)|May 29, 2023

Topic ModelingComputer Science271 citations

TL;DR

Direct Preference Optimization (DPO) directly optimizes language model policies from human preferences without explicit reward modeling or RL. It achieves comparable or better alignment than PPO-based RLHF and is simpler to implement and train.

ABSTRACT

While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.

Motivation & Objective

• Motivate the need for steerable, safe, and aligned large language models by leveraging human preferences.
• Propose a new paradigm to optimize policies directly from preferences without explicit reward modeling or RL.
• Show that a reparameterization yields a closed-form optimal policy, enabling a simple classification loss.
• Compare DPO to PPO-based RLHF across tasks like sentiment control, summarization, and dialogue.
• Demonstrate stability, efficiency, and scalability of DPO on models up to 6B parameters.

Proposed method

• Introduce a reward-model parameterization that allows extracting the optimal policy in closed form (Eq. 4).
• Reparameterize the reward via r(x,y)=β log(π(y|x)/π_ref(y|x)) and derive a Bradley-Terry-based preference loss over policy outputs (Eq. 7).
• Formulate a binary cross-entropy objective over preferred/dispreferred pairs using the implicit reward (Eq. 7).
• Weight the loss by the implicit reward-ordering error to prevent degeneration (gradient form discussed).
• Outline a practical DPO pipeline: sample from π_ref, collect human preferences, and optimize with the DPO loss.
• Discuss theoretical properties showing equivalence to reward-based RL under Plackett-Luce/Bradley-Terry models and robustness advantages over actor-critic methods.

Figure 1: DPO optimizes for human preferences while avoiding reinforcement learning. Existing methods for fine-tuning language models with human feedback first fit a reward model to a dataset of prompts and human preferences over pairs of responses, and then use RL to find a policy that maximizes th

Experimental results

Research questions

• RQ1 Can direct optimization of policy from human preferences match or surpass RLHF with PPO across tasks?
• RQ2 Does the DPO reparameterization recover the optimal policy without explicit reward modeling or RL loops?
• RQ3 How does DPO compare to baseline methods in sentiment control, summarization, and dialogue in terms of performance and stability?
• RQ4 Is DPO more efficient and robust to hyperparameters and temperature than PPO-based RLHF?
• RQ5 What theoretical guarantees under Plackett-Luce/Bradley-Terry models support DPO’s validity?

Key findings

• DPO achieves the highest reward for a given KL constraint frontier, dominating PPO on the sentiment frontier.
• DPO matches or surpasses PPO-based RLHF in summarization and dialogue tasks with less hyperparameter tuning.
• DPO remains robust across sampling temperatures and converges quickly to strong performance.
• In controlled sentiment settings, DPO outperforms PPO even when PPO has access to ground-truth rewards.
• DPO provides competitive performance with up to 6B-parameter LMs using a simple, Stable training objective.

Figure 2: Left. The frontier of expected reward vs KL to the reference policy. DPO provides the highest expected reward for all KL values, demonstrating the quality of the optimization. Right. TL;DR summarization win rates vs. human-written summaries, using GPT-4 as evaluator. DPO exceeds PPO’s best