[Paper Review] Fine-Tuning Language Models from Human Preferences

Daniel M. Ziegler, Nisan Stiennon+6 more|arXiv (Cornell University)|Sep 18, 2019

Topic Modeling46 references383 citations

TL;DR

The paper trains reward models from human preferences and uses RL with a KL constraint to fine-tune large language models for stylized text continuation and abstractive summarization, achieving strong human-judgment-based performance with relatively little labeled data.

ABSTRACT

Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks. In this paper, we build on advances in generative pretraining of language models to apply reward learning to four natural language tasks: continuing text with positive sentiment or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic continuation we achieve good results with only 5,000 comparisons evaluated by humans. For summarization, models trained with 60,000 comparisons copy whole sentences from the input but skip irrelevant preamble; this leads to reasonable ROUGE scores and very good performance according to our human labelers, but may be exploiting the fact that labelers rely on simple heuristics.

Motivation & Objective

• Demonstrate reward learning from human preferences for language tasks.
• Combine pretrained language models with RL to optimize human-judged rewards.
• Explore online vs offline data collection for reward modeling.
• Assess stylistic continuation and summarization performance using human evaluations.

Proposed method

• Start from a pretrained language model (GPT-2 774M) and a reward model r trained from human comparisons across four candidate continuations.
• Train r using a softmax cross-entropy loss over human selections (Eq. 1).
• Fine-tune the policy π with Proximal Policy Optimization (PPO) using a modified reward R(x,y)=r(x,y)−β log(π(y|x)/ρ(y|x)).
• Impose a KL constraint between the updated policy π and the initial model ρ to prevent drift (β term).
• Optionally vary β online to target a desired KL(π,ρ).
• Collect human labels in online or offline data collection modes and retrain r periodically (online) or once (offline).
• For stylistic continuation tasks, optimize sentiment and descriptiveness; for summarization, optimize CNN/Daily Mail and TL;DR datasets.

Experimental results

Research questions

• RQ1 Can a reward model learned from human preferences effectively guide RL fine-tuning of large language models?
• RQ2 How does online vs offline reward data collection impact performance and stability?
• RQ3 What are the qualitative and quantitative differences when optimizing for stylistic continuation versus summarization?
• RQ4 To what extent does a KL constraint influence fidelity, coherence, and adherence to style during RL fine-tuning?

Key findings

• A small amount of human feedback (as low as ~5k comparisons) can bias the model toward human-preferred stylistic continuations.
• Reward-model-based RL improves human-perceived quality on stylistic continuation versus zero-shot or purely supervised baselines.
• For summarization, models trained with 60k human comparisons behave like “smart copy-ers,” often copying whole sentences, and achieve favorable human judgments compared to baselines (though may exploit copying heuristics).
• Online data collection generally improves performance for summarization, while stylistic tasks show similar gains with offline data.
• RL fine-tuning starting from supervised baselines often yields strong ROUGE scores, but human evaluations favor RL-tuned policies that optimize actual human preferences.
• The 60k RL-tuned models copy sentences heavily (71% TL;DR, 98% CNN/DM for copied sentences), indicating a strong extractive tendency under preference-driven RL.