[Paper Review] Learning to summarize from human feedback

Nisan Stiennon|arXiv (Cornell University)|Sep 2, 2020

Topic Modeling71 references37 citations

TL;DR

The paper trains a reward model from human comparisons to fine-tune a summarization policy with reinforcement learning, achieving higher-quality summaries than supervised baselines on TL;DR and transferring well to CNN/DM without domain-specific fine-tuning.

ABSTRACT

As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about -- summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles, producing summaries nearly as good as the human reference without any news-specific fine-tuning. We conduct extensive analyses to understand our human feedback dataset and fine-tuned models We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want.

Motivation & Objective

• Motivate aligning language-model summarization with human judgments rather than simple reference-based metrics.
• Develop a batch offline pipeline using human comparisons to train a reward model.
• Fine-tune a summarization policy with PPO to maximize human-preference rewards.
• Evaluate generalization to new domains and analyze reward-model behavior.
• Publicly release the dataset and code to advance research on human-in-the-loop summarization.

Proposed method

• Collect a large batch of human comparisons between candidate summaries for Reddit TL;DR posts.
• Train a reward model to predict human preference between two summaries for a given post.
• Fine-tune a GPT-3–style Transformer policy with PPO to maximize the reward model output, including a KL penalty to stay near the supervised baseline.
• Use an offline, batch-based loop: collect comparisons, train reward model, optimize policy, then sample from the new policy to gather more data.
• Evaluate transfer to CNN/DM without news-domain fine-tuning and compare to ROUGE and reference summaries.
• Release a 64k+ human-feedback dataset and provide inference code and model cards.

Experimental results

Research questions

• RQ1 Can human-preference–trained reward models produce summaries preferred by humans over reference summaries and larger supervised models?
• RQ2 Do reward-model–driven policies generalize across domains (e.g., TL;DR to CNN/DailyMail) without task-specific fine-tuning?
• RQ3 How does the reward model's quality and scale affect the final summaries and their alignment with human judgments?
• RQ4 Is optimizing against ROUGE or other automatic metrics a reliable proxy for human preferences?
• RQ5 What are the limitations and risks of training with human feedback for summarization tasks?

Key findings

• Policies trained with human feedback outperform strong supervised baselines on TL;DR in human preference tests.
• The 1.3B and 6.7B human-feedback models outperform their supervised counterparts, with larger gains at scale.
• Human-feedback models transfer to CNN/DM with near-reference quality without news-domain fine-tuning.
• Reward models better predict human preferences than ROUGE or log probability measures, and optimizing against them yields better summaries.
• Reward-model performance improves with more data and larger model size, with diminishing returns.
• The dataset of 64,832 TL;DR comparisons and related evaluation data are publicly released.