[Paper Review] A Deep Reinforced Model for Abstractive Summarization

Romain Paulus, Caiming Xiong+1 more|arXiv (Cornell University)|May 11, 2017

Topic Modeling39 references1,273 citations

TL;DR

The paper introduces a neural abstractive summarization model with intra-attention (over input and generated output) and a hybrid learning objective combining supervised cross-entropy with reinforcement learning, achieving state-of-the-art ROUGE on CNN/Daily Mail and strong results on NYT, plus improved readability.

ABSTRACT

Attentional, RNN-based encoder-decoder models for abstractive summarization have achieved good performance on short input and output sequences. For longer documents and summaries however these models often include repetitive and incoherent phrases. We introduce a neural network model with a novel intra-attention that attends over the input and continuously generated output separately, and a new training method that combines standard supervised word prediction and reinforcement learning (RL). Models trained only with supervised learning often exhibit "exposure bias" - they assume ground truth is provided at each step during training. However, when standard word prediction is combined with the global sequence prediction training of RL the resulting summaries become more readable. We evaluate this model on the CNN/Daily Mail and New York Times datasets. Our model obtains a 41.16 ROUGE-1 score on the CNN/Daily Mail dataset, an improvement over previous state-of-the-art models. Human evaluation also shows that our model produces higher quality summaries.

Motivation & Objective

• Address the challenge of long-document abstractive summarization with repetitive phrases.
• Develop an intra-attentive encoder-decoder architecture that tracks attention over input and generated output to reduce repetition.
• Mitigate exposure bias by combining supervised learning with reinforcement learning in a mixed objective.
• Improve readability and coherence of long-form summaries beyond purely maximum-likelihood training.

Proposed method

• Introduce intra-temporal attention on the input sequence to diversify focus across decoding steps.
• Add intra-decoder attention to track previously generated tokens and reduce repetition.
• Incorporate a pointer-generator mechanism to copy rare or unseen words from the source.
• Share decoder and embedding weights to enhance output generation.
• Apply a self-critical reinforcement learning objective (and a mixed ML/RL objective) to optimize ROUGE while maintaining readability.
• Impose a test-time constraint to avoid outputting repeating trigrams in beam search.

Experimental results

Research questions

• RQ1 Can intra-temporal input attention and intra-decoder attention reduce repetition in long abstractive summaries?
• RQ2 Does combining supervised learning with reinforcement learning (and a mixed objective) improve both ROUGE scores and readability for long summaries?
• RQ3 How does the model perform on CNN/Daily Mail and the NYT abstractive summarization tasks compared to prior work?
• RQ4 What is the impact of these techniques on human judgments of readability and relevance?

Key findings

• The model achieves 41.16 ROUGE-1 on CNN/Daily Mail, surpassing prior state-of-the-art baselines.
• Reinforcement learning (RL) improves ROUGE scores but can reduce readability, while a mixed ML+RL objective yields higher readability and competitive ROUGE.
• Intra-decoder attention improves ROUGE-1 on CNN/Daily Mail for longer ground-truth summaries, but not on NYT, indicating benefits depend on output length.
• On NYT, the ML+RL with intra-attention setup attains strong ROUGE measures, outperforming several extractive baselines and prior abstractive models.
• Human evaluation shows RL alone yields lower readability, while ML+RL achieves the best readability and relevance balance.
• The combination of intra-attention and mixed training is particularly beneficial for long-sequence summarization and can be extended to other long-sequence generation tasks.