[Paper Review] Recursively Summarizing Books with Human Feedback

Jeff Wu, Long Ouyang+5 more|arXiv (Cornell University)|Sep 22, 2021

Topic Modeling66 references66 citations

TL;DR

The paper combines recursive task decomposition with learning from human feedback to train a model that can summarize entire books by iteratively summarizing smaller parts and recursively composing them.

ABSTRACT

A major challenge for scaling machine learning is training models to perform tasks that are very difficult or time-consuming for humans to evaluate. We present progress on this problem on the task of abstractive summarization of entire fiction novels. Our method combines learning from human feedback with recursive task decomposition: we use models trained on smaller parts of the task to assist humans in giving feedback on the broader task. We collect a large volume of demonstrations and comparisons from human labelers, and fine-tune GPT-3 using behavioral cloning and reward modeling to do summarization recursively. At inference time, the model first summarizes small sections of the book and then recursively summarizes these summaries to produce a summary of the entire book. Our human labelers are able to supervise and evaluate the models quickly, despite not having read the entire books themselves. Our resulting model generates sensible summaries of entire books, even matching the quality of human-written summaries in a few cases ($\sim5\%$ of books). We achieve state-of-the-art results on the recent BookSum dataset for book-length summarization. A zero-shot question-answering model using these summaries achieves state-of-the-art results on the challenging NarrativeQA benchmark for answering questions about books and movie scripts. We release datasets of samples from our model.

Motivation & Objective

• Address the challenge of training models on long, difficult-to-evaluate tasks via scalable oversight.
• Develop a recursive summarization framework that decomposes book text into a tree of subtasks.
• Train a single model end-to-end using behavioral cloning and reward modeling from human preferences.
• Show that recursion enables handling books of arbitrary length and compare to baselines on book-length datasets.

Proposed method

• Decompose long texts into a tree of summarization tasks where leaf tasks summarize passages (height-0).
• Train a model to perform leaf summarization via behavioral cloning and to compose higher-level summaries via outputs of lower-level tasks.
• Use human comparisons to train a reward model and apply RL with KL regularization to optimize summaries.
• Recursively generate inputs by composing summaries, conditioning on prior context from same depth to maintain coherence.
• Implement a training curriculum to mitigate auto-induced distributional shift and focus on early tree levels before broader tree levels.
• Evaluate on full books unseen during training, BookSum, and NarrativeQA, and release datasets.

Experimental results

Research questions

• RQ1 Can recursive task decomposition combined with human feedback scale to long-form abstractive summarization of books?
• RQ2 How does RL with human preferences compare to BC on leaf tasks and full-tree summarization?
• RQ3 What are the efficiency gains from using comparisons versus demonstrations in this setup?
• RQ4 Do summaries generated from hierarchical leaves generalize to entire books and to downstream QA benchmarks?

Key findings

• RL-trained models on the full tree outperform BC baselines at large model sizes, though still below human performance on average.
• First-subtree RL models match or exceed full-tree RL performance, showing good generalization from early tree levels.
• 175B RL models achieve state-of-the-art results on BookSum and competitive results on NarrativeQA when using book-derived summaries as input.
• Depth-0 and depth-1 summaries can be combined to produce coherent full-book summaries via recursion, with some summaries approaching human quality (≈5% at top size).
• RL on comparisons is more efficient than BC on demonstrations in terms of human time for data collection, especially at higher data budgets.
• The approach enables answering questions about books using zero-shot QA with summaries as input, achieving competitive results on NarrativeQA.