[Paper Review] Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data

Fahim Tajwar, Anikait Singh+7 more|arXiv (Cornell University)|Apr 22, 2024

Efficiency Analysis Using DEADecision Sciences3 citations

TL;DR

This paper demonstrates that preference fine-tuning of large language models (LLMs) benefits significantly from using suboptimal, on-policy responses and negative gradient objectives, which enable faster and more effective redistribution of probability mass to high-reward responses. The authors introduce the concept of 'mode-seeking' objectives—unified under reverse KL divergence—that outperform standard maximum likelihood and offline supervised methods, especially when high-reward responses lie in low-probability regions of the reference policy.

ABSTRACT

Learning from preference labels plays a crucial role in fine-tuning large language models. There are several distinct approaches for preference fine-tuning, including supervised learning, on-policy reinforcement learning (RL), and contrastive learning. Different methods come with different implementation tradeoffs and performance differences, and existing empirical findings present different conclusions, for instance, some results show that online RL is quite important to attain good fine-tuning results, while others find (offline) contrastive or even purely supervised methods sufficient. This raises a natural question: what kind of approaches are important for fine-tuning with preference data and why? In this paper, we answer this question by performing a rigorous analysis of a number of fine-tuning techniques on didactic and full-scale LLM problems. Our main finding is that, in general, approaches that use on-policy sampling or attempt to push down the likelihood on certain responses (i.e., employ a "negative gradient") outperform offline and maximum likelihood objectives. We conceptualize our insights and unify methods that use on-policy sampling or negative gradient under a notion of mode-seeking objectives for categorical distributions. Mode-seeking objectives are able to alter probability mass on specific bins of a categorical distribution at a fast rate compared to maximum likelihood, allowing them to relocate masses across bins more effectively. Our analysis prescribes actionable insights for preference fine-tuning of LLMs and informs how data should be collected for maximal improvement.

Motivation & Objective

• To resolve confusion in the LLM community about whether on-policy RL or offline methods are necessary for preference fine-tuning.
• To investigate the role of on-policy data collection and negative gradient objectives in improving fine-tuning performance.
• To unify diverse methods like on-policy RL and contrastive learning under a common theoretical framework.
• To identify when and why on-policy sampling and negative gradients are particularly effective for preference alignment.
• To provide actionable insights for data collection and optimization in LLM preference fine-tuning.

Proposed method

• The authors design didactic bandit problems, synthetic LLM problems, and full-scale LLM experiments using AlpacaFarm and UltraFeedback datasets.
• They compare methods including supervised learning, contrastive learning, and on-policy reinforcement learning under controlled coverage and geometric conditions.
• They introduce a theoretical framework based on reverse KL divergence to explain why negative gradient objectives are more effective at redistributing probability mass.
• They formalize 'mode-seeking' behavior as the ability to rapidly concentrate probability mass on high-reward response bins, contrasting it with 'mode-covering' maximum likelihood objectives.
• They analyze the behavior of forward and reverse KL divergences to show that reverse KL prioritizes regions of higher current likelihood, enabling faster reweighting.
• They empirically validate that on-policy sampling and negative gradients are complementary and especially critical when high-reward responses are rare under the reference policy.

Experimental results

Research questions

• RQ1 When is on-policy sampling necessary for effective preference fine-tuning, and when can offline methods suffice?
• RQ2 Why do methods with negative gradients (e.g., contrastive or RL objectives) outperform standard supervised fine-tuning with maximum likelihood?
• RQ3 How do geometric relationships between the reference policy and the true reward peak affect method performance?
• RQ4 Can a unifying principle explain the success of on-policy RL and contrastive learning in preference fine-tuning?
• RQ5 What role does the initial likelihood of high-reward responses under the reference policy play in determining the effectiveness of different fine-tuning strategies?

Key findings

• On-policy sampling significantly improves performance when high-reward responses lie in low-likelihood regions of the reference policy, as it enables direct exploration of these regions.
• Negative gradient objectives—such as those in contrastive learning or on-policy RL—outperform maximum likelihood and offline supervised methods by enabling faster probability mass redistribution to high-reward responses.
• The combination of on-policy sampling and negative gradients provides complementary benefits, especially when preference data is skewed away from the reference policy.
• Methods that induce mode-seeking behavior via reverse KL divergence can concentrate probability mass on a subset of high-reward responses much faster than mode-covering objectives.
• Supervised fine-tuning methods like Pref-FT and Binary Feed-ME fail to effectively shift mass from low-reward to high-reward responses, even with full data coverage.
• Theoretical analysis shows that reverse KL prioritizes regions with higher current likelihood, making it more efficient at reweighting probability mass in a single step.