[Paper Review] Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints

Chaoqi Wang, Yibo Jiang+3 more|arXiv (Cornell University)|Sep 28, 2023

Data Management and AlgorithmsComputer Science3 citations

TL;DR

This paper introduces f-DPO, a generalized framework for Direct Preference Optimization (DPO) that extends beyond reverse KL divergence by incorporating diverse f-divergences such as Jensen-Shannon, forward KL, and α-divergences. By rigorously solving the Karush–Kuhn–Tucker (KKT) conditions—particularly complementary slackness—it eliminates the need to estimate the normalizing constant in the Bradley-Terry model, enabling a tractable, analytical mapping between reward functions and optimal policies. The method achieves superior alignment efficiency and improved calibration error control compared to PPO-based approaches, while balancing alignment performance and generation diversity.

ABSTRACT

The increasing capabilities of large language models (LLMs) raise opportunities for artificial general intelligence but concurrently amplify safety concerns, such as potential misuse of AI systems, necessitating effective AI alignment. Reinforcement Learning from Human Feedback (RLHF) has emerged as a promising pathway towards AI alignment but brings forth challenges due to its complexity and dependence on a separate reward model. Direct Preference Optimization (DPO) has been proposed as an alternative, and it remains equivalent to RLHF under the reverse KL regularization constraint. This paper presents $f$-DPO, a generalized approach to DPO by incorporating diverse divergence constraints. We show that under certain $f$-divergences, including Jensen-Shannon divergence, forward KL divergences and $α$-divergences, the complex relationship between the reward and optimal policy can also be simplified by addressing the Karush-Kuhn-Tucker conditions. This eliminates the need for estimating the normalizing constant in the Bradley-Terry model and enables a tractable mapping between the reward function and the optimal policy. Our approach optimizes LLMs to align with human preferences in a more efficient and supervised manner under a broad set of divergence constraints. Empirically, adopting these divergences ensures a balance between alignment performance and generation diversity. Importantly, $f$-DPO outperforms PPO-based methods in divergence efficiency, and divergence constraints directly influence expected calibration error (ECE).

Motivation & Objective

• To address the limitations of existing DPO and RLHF methods that rely exclusively on reverse KL divergence, which can reduce generation diversity and limit model expressiveness.
• To generalize DPO beyond reverse KL by incorporating a broad class of f-divergences, including forward KL, Jensen-Shannon, and α-divergences, to enable richer policy optimization under varied constraints.
• To eliminate the need for estimating the normalizing constant in the Bradley-Terry model by analytically resolving the KKT conditions, particularly complementary slackness, under specific divergence families.
• To achieve more efficient, stable, and supervised fine-tuning of LLMs for human preference alignment without requiring reward models or reinforcement learning.
• To empirically validate that diverse divergence constraints improve the balance between alignment performance and generation diversity, and directly influence expected calibration error (ECE).

Proposed method

• The method generalizes DPO by formulating the preference optimization problem under a family of f-divergence constraints, including Jensen-Shannon, forward KL, and α-divergences with α ∈ (0,1).
• It derives an analytical solution to the KKT conditions of the constrained optimization problem, particularly leveraging complementary slackness to eliminate the intractable normalizing constant in the Bradley-Terry model.
• The approach establishes a closed-form mapping between the reward function and the optimal policy under these divergences, enabling direct optimization without iterative RL or reward model training.
• The framework supports end-to-end supervised fine-tuning of LLMs using preference data, with divergence constraints explicitly shaping the policy update to balance alignment and diversity.
• It enables the use of multiple divergence types as hyperparameters, allowing practitioners to tune for specific trade-offs between safety, diversity, and performance.
• The method is theoretically grounded in convex optimization and f-divergence theory, with a focus on tractability and interpretability in the context of LLM alignment.

Experimental results

Research questions

• RQ1 Can the DPO framework be generalized beyond reverse KL divergence to include other f-divergences such as forward KL, Jensen-Shannon, and α-divergences?
• RQ2 Does the elimination of the normalizing constant in the Bradley-Terry model remain feasible under these generalized divergences via KKT condition analysis?
• RQ3 Can the resulting f-DPO framework achieve better alignment efficiency and diversity balance compared to PPO-based RLHF methods?
• RQ4 How do different divergence constraints affect the expected calibration error (ECE) of the fine-tuned LLM?
• RQ5 Can the generalized f-DPO approach maintain strong alignment performance while enabling more flexible and interpretable policy optimization?

Key findings

• f-DPO successfully generalizes DPO to multiple f-divergences, including Jensen-Shannon, forward KL, and α-divergences with α ∈ (0,1), by analytically resolving the KKT conditions and eliminating the need for normalizing constant estimation.
• The method achieves a direct, analytical mapping between the reward function and the optimal policy under these divergences, enabling efficient, supervised fine-tuning without reward models or reinforcement learning.
• Empirically, f-DPO outperforms PPO-based methods in divergence efficiency, demonstrating faster convergence and better alignment performance under the same preference data.
• Different divergence constraints directly influence the expected calibration error (ECE), with f-DPO enabling explicit control over model calibration through divergence selection.
• The use of diverse divergences leads to a better balance between alignment performance and generation diversity, mitigating the mode-seeking behavior of reverse KL and improving model expressiveness.
• The framework enables a broader spectrum of modeling behaviors—such as mass-covering or mode-seeking—depending on the chosen divergence, offering greater flexibility for application-specific alignment.