[Paper Review] SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents

Ramesh Nallapati, Feifei Zhai+1 more|arXiv (Cornell University)|Nov 14, 2016

Topic Modeling749 citations

TL;DR

SummaRuNNer is a two-layer bi-directional GRU-based sequence classifier for extractive document summarization that yields state-of-the-art or competitive results and offers interpretable analysis of its decisions. It also introduces abstractive training to learn from abstractive summaries without sentence-level labels.

ABSTRACT

We present SummaRuNNer, a Recurrent Neural Network (RNN) based sequence model for extractive summarization of documents and show that it achieves performance better than or comparable to state-of-the-art. Our model has the additional advantage of being very interpretable, since it allows visualization of its predictions broken up by abstract features such as information content, salience and novelty. Another novel contribution of our work is abstractive training of our extractive model that can train on human generated reference summaries alone, eliminating the need for sentence-level extractive labels.

Motivation & Objective

• Develop a neural, end-to-end model for sentence-level extractive summarization of single documents.
• Improve performance to be better than or comparable to state-of-the-art extractive methods on large datasets.
• Provide interpretability by attributing decisions to explicit abstract features such as content, salience, and novelty.
• Propose abstractive training to enable learning from abstractive summaries without sentence-level extractive labels.

Proposed method

• Model the task as a sequential sentence-level binary classification using a two-layer bi-directional GRU RNN.
• Bottom word-level GRU captures intra-sentence representations; a top sentence-level bi-GRU encodes sentence representations for document-level context.
• Sentence classification uses a logistic layer with features for information content, salience, novelty, absolute and relative position.
• Summary state s_j is a running weighted sum of past sentence representations, guiding redundancy and relevance during prediction.
• Train with negative log-likelihood on extractive labels; derive extractive labels from abstractive summaries via a greedy Rouge-based optimization (abstractive training) or use ground-truth extractive labels (extractive training).
• When abstractively trained, couple a decoder to model reference summary words and backpropagate through the summary representation to influence extractive probabilities.

Experimental results

Research questions

• RQ1 Can a recurrent neural network-based sequence classifier effectively perform single-document extractive summarization in an end-to-end fashion?
• RQ2 Does incorporating content, salience, novelty, and positional features improve extractive summary quality compared to baselines?
• RQ3 Can abstractive training signals be used to train an extractive model without sentence-level extractive labels, while maintaining performance?
• RQ4 How does SummaRuNNer perform relative to state-of-the-art models on large-scale CNN/Daily Mail and DUC 2002 datasets?
• RQ5 What interpretability insights can be gained from the explicit feature-driven decision process?

Key findings

• SummaRuNNer achieves performance that is better than or comparable to state-of-the-art extractive models on benchmarks.
• Extractive training yields statistically significant improvements over a competitive abstractive setup on the Daily Mail dataset with limited-length Rouge measures.
• On the CNN/Daily Mail corpus, SummaRuNNer significantly outperforms an abstractive encoder–decoder model on full-length Rouge metrics.
• The model provides interpretable visualization of decisions via separate content, salience, novelty, and positional components in the classification score.
• Abstractive training reduces the need for sentence-level extractive labels but tends to lag the extractive-trained model in most datasets examined.
• On out-of-domain DUC 2002 data, SummaRuNNer is on par with a comparable model but generally behind graph-based approaches like TGRAPH and URANK, highlighting domain robustness concerns for deep learning extractive methods.