[Paper Review] Understanding the performance gap between online and offline alignment algorithms

Yunhao Tang, Daniel Guo+9 more|arXiv (Cornell University)|May 14, 2024

Genomics and Phylogenetic StudiesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper investigates why online reinforcement learning from human feedback (RLHF) outperforms offline alignment methods in large language model alignment, despite offline methods being computationally cheaper. Through controlled ablation studies, it identifies on-policy data generation as the key factor behind online methods' superior generative performance, revealing a fundamental trade-off between discriminative and generative capabilities that offline algorithms fail to resolve.

ABSTRACT

Reinforcement learning from human feedback (RLHF) is the canonical framework for large language model alignment. However, rising popularity in offline alignment algorithms challenge the need for on-policy sampling in RLHF. Within the context of reward over-optimization, we start with an opening set of experiments that demonstrate the clear advantage of online methods over offline methods. This prompts us to investigate the causes to the performance discrepancy through a series of carefully designed experimental ablations. We show empirically that hypotheses such as offline data coverage and data quality by itself cannot convincingly explain the performance difference. We also find that while offline algorithms train policy to become good at pairwise classification, it is worse at generations; in the meantime the policies trained by online algorithms are good at generations while worse at pairwise classification. This hints at a unique interplay between discriminative and generative capabilities, which is greatly impacted by the sampling process. Lastly, we observe that the performance discrepancy persists for both contrastive and non-contrastive loss functions, and appears not to be addressed by simply scaling up policy networks. Taken together, our study sheds light on the pivotal role of on-policy sampling in AI alignment, and hints at certain fundamental challenges of offline alignment algorithms.

Motivation & Objective

• To investigate the root causes of the performance gap between online and offline alignment algorithms in large language model alignment.
• To evaluate whether data coverage, data quality, optimization dynamics, loss functions, or model scaling explain the observed performance discrepancy.
• To determine whether offline algorithms can achieve parity with online methods through improved data generation or architectural scaling.
• To understand the interplay between discriminative (pairwise classification) and generative (text generation) capabilities in policy optimization.

Proposed method

• Used KL divergence between the RLHF policy and the supervised fine-tuned (SFT) policy as a unified budget metric to fairly compare online and offline algorithms.
• Conducted controlled ablation experiments to test hypotheses related to data distribution, quality, optimization procedure, and loss functions.
• Compared online algorithms (e.g., PPO) with offline algorithms (e.g., DPO) on open-source datasets under identical KL budget constraints.
• Evaluated both contrastive and non-contrastive loss functions to assess their impact on the performance gap.
• Generated synthetic offline datasets with distributional proximity to the initial policy to simulate on-policy data collection.
• Measured performance via both pairwise classification accuracy (discriminative) and generation quality (generative) metrics.

Experimental results

Research questions

• RQ1 Why do online alignment algorithms consistently outperform offline algorithms under the same KL budget?
• RQ2 To what extent do data coverage and data quality explain the performance gap between online and offline methods?
• RQ3 Is there a fundamental trade-off between discriminative and generative capabilities in offline policy optimization?
• RQ4 Does the performance gap persist across different loss functions (contrastive vs. non-contrastive) and model scaling?
• RQ5 Can the performance of offline algorithms be improved by generating data that mimics on-policy distribution?

Key findings

• Online algorithms consistently outperform offline algorithms across all KL budget levels, achieving higher peak performance on both generation and classification tasks.
• Offline data coverage and data quality alone cannot explain the performance gap, as even high-quality, high-coverage datasets fail to close the gap.
• Offline policies are better at pairwise classification but significantly worse at text generation, indicating a strong disconnect between discriminative and generative performance.
• Online policies show superior generative quality due to on-policy sampling, which continuously shifts the distribution and improves generation, a mechanism absent in static offline data.
• The performance gap persists across both contrastive and non-contrastive loss functions, indicating it is not an artifact of specific optimization objectives.
• Scaling up policy networks does not resolve the performance gap, suggesting that architectural capacity alone is insufficient to overcome the limitations of offline learning.