[Paper Review] Neural Summarization by Extracting Sentences and Words

Jianpeng Cheng, Mirella Lapata|arXiv (Cornell University)|Mar 23, 2016

Topic Modeling34 references128 citations

TL;DR

The paper presents a data-driven, encoder–extractor framework for single-document summarization that can extract either sentences or words using neural networks and hierarchical representations, trained on large DailyMail-derived datasets.

ABSTRACT

Traditional approaches to extractive summarization rely heavily on human-engineered features. In this work we propose a data-driven approach based on neural networks and continuous sentence features. We develop a general framework for single-document summarization composed of a hierarchical document encoder and an attention-based extractor. This architecture allows us to develop different classes of summarization models which can extract sentences or words. We train our models on large scale corpora containing hundreds of thousands of document-summary pairs. Experimental results on two summarization datasets demonstrate that our models obtain results comparable to the state of the art without any access to linguistic annotation.

Motivation & Objective

• Motivate data-driven extractive summarization without hand-engineered features.
• Propose a hierarchical neural encoder for documents and an attention-based extractor for sentences or words.
• Show that neural extractive and extractive-generated summaries can match or approach state-of-the-art systems on standard benchmarks.
• Demonstrate training on large-scale, automatically labeled datasets derived from DailyMail highlights.

Proposed method

• Use a hierarchical document reader: convolutional sentence encoder with max-over-time pooling to produce sentence vectors, followed by an LSTM-based document encoder over sentence vectors.
• Introduce a sentence extractor that uses attention-driven labeling of sentences using encoder and extractor hidden states, with curriculum learning to mitigate training/testing mismatch.
• Develop a word extractor that performs generation over a restricted vocabulary drawn from the document, using hierarchical attention to select next word from input content.
• Train two large-scale datasets from DailyMail: a sentence-extraction dataset by labeling sentences whose overlap with highlights is high, and a word-extraction dataset by matching highlight words to article content (with substitutions for OOV terms).
• Evaluate with ROUGE on DUC-2002 and DailyMail test sets, plus human judgments; compare against baselines (lead, logistic regression, ILP, tgraph, urank) and an abstractive baseline.

Experimental results

Research questions

• RQ1 Can a neural encoder–extractor architecture effectively perform extractive summarization at the sentence level without linguistic annotations?
• RQ2 Can the approach be extended to word-level extraction with a generation-style decoder constrained to the document vocabulary?
• RQ3 How do neural sentence-extraction and word-extraction models compare to traditional feature-based and graph/ILP-based methods on standard benchmarks?
• RQ4 What is the impact of training data size and quality (large DailyMail-derived corpora) on performance compared to small, annotated datasets?
• RQ5 Does a hierarchical representation (sentence CNN encoder + document-level RNN) improve saliency detection and summary coherence?

Key findings

• Sentence-extraction model nn-se achieves competitive Rouge scores on DUC-2002, outperforming several baselines and approaching state-of-the-art without linguistic features.
• Word-extraction model nn-we consistently outperforms the abstractive baseline and shows robustness for proper nouns and rare words by copying from the source document.
• Both neural extractors (nn-se and nn-we) generally outperform or closely match strong baselines (lead, lreg, ilp, tgraph, urank) on DUC-2002; in DailyMail, nn-se remains strongest across Rouge metrics.
• Human evaluation ranks nn-se near the top, with substantial preference over lead and several baselines, and approaching the human gold standard.
• The open-vocabulary abstractive model nn-abs performs worse than the extractive-word model nn-we, highlighting the benefits of restricted-vocabulary extraction for accuracy and coherence.