[Paper Review] ORPO: Monolithic Preference Optimization without Reference Model

Jiwoo Hong, Noah Lee+1 more|arXiv (Cornell University)|Mar 12, 2024

Multi-Criteria Decision Making7 citations

TL;DR

ORPO introduces a reference-free, monolithic odds-ratio based preference optimization method that improves fine-tuning for alignment without requiring RLHF or a reference model, achieving strong instruction-following results across multiple models and datasets.

ABSTRACT

While recent preference alignment algorithms for language models have demonstrated promising results, supervised fine-tuning (SFT) remains imperative for achieving successful convergence. In this paper, we study the crucial role of SFT within the context of preference alignment, emphasizing that a minor penalty for the disfavored generation style is sufficient for preference-aligned SFT. Building on this foundation, we introduce a straightforward and innovative reference model-free monolithic odds ratio preference optimization algorithm, ORPO, eliminating the necessity for an additional preference alignment phase. We demonstrate, both empirically and theoretically, that the odds ratio is a sensible choice for contrasting favored and disfavored styles during SFT across the diverse sizes from 125M to 7B. Specifically, fine-tuning Phi-2 (2.7B), Llama-2 (7B), and Mistral (7B) with ORPO on the UltraFeedback alone surpasses the performance of state-of-the-art language models with more than 7B and 13B parameters: achieving up to 12.20% on $ ext{AlpacaEval}_{2.0}$ (Figure 1), 66.19% on IFEval (instruction-level loose, Table 6), and 7.32 in MT-Bench (Figure 12). We release code and model checkpoints for Mistral-ORPO-$α$ (7B) and Mistral-ORPO-$β$ (7B).

Motivation & Objective

• Investigate the role of supervised fine-tuning (SFT) in preference alignment.
• Propose a reference-model-free monolithic alignment method (ORPO).
• Demonstrate ORPO's effectiveness across model sizes from 125M to 7B on standard benchmarks.
• Compare ORPO against RLHF, DPO, and SFT baselines in various tasks.

Proposed method

• Define an odds-ratio based penalty appended to the negative log-likelihood loss.
• Combine L_SFT with a relative ratio loss L_OR to form L_ORPO.
• Use log-sigmoid of the log odds ratio to stabilize optimization.
• Evaluate on datasets HH-RLHF and UltraFeedback with models Phi-2, Llama-2, and Mistral.
• Compare against SFT, PPO, and DPO across model scales.

Experimental results

Research questions

• RQ1 Can SFT alone be sufficient for preference alignment with a simple penalty term?
• RQ2 Does a reference-model-free odds-ratio objective improve alignment performance across multiple model sizes?
• RQ3 How does ORPO compare to RLHF and DPO in terms of win rate and reward distribution on standard benchmarks?
• RQ4 What is the impact of ORPO on instruction-following capabilities and multi-turn tasks?
• RQ5 Is ORPO more computationally efficient than reference-based methods?

Key findings

• ORPO achieves strong instruction-following performance, surpassing some 7B+ state-of-the-art models on AlpacaEval 2.0 and MT-Bench.
• Mistral-ORPO-α and Mistral-ORPO-β (7B) reach 11.33% and 12.20% on AlpacaEval2.0, and 7.23 and 7.32 on MT-Bench.
• ORPO outperforms SFT and PPO across all tested model scales on HH-RLHF, with win rates up to 78.0% against SFT and 79.4% against PPO.
• On UltraFeedback, ORPO shows win rates up to 80.5% against SFT and 85.8% against PPO, with larger models showing stronger DPO comparison.
• ORPO does not require a reference model, reducing forward passes and computational cost relative to RLHF/DPO.
• Reward distributions indicate higher expected rewards for ORPO compared to RLHF and DPO in the tested setups.