[Paper Review] Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization

Shashi Narayan, Shay B. Cohen+1 more|arXiv (Cornell University)|Aug 27, 2018

Topic Modeling25 references35 citations

TL;DR

This paper introduces extreme summarization, a new single-document summarization task requiring one-sentence abstractive summaries that answer 'What is the article about?'. It proposes a topic-aware convolutional sequence-to-sequence model (T-ConvS2S) based entirely on CNNs, which outperforms extractive and RNN-based abstractive models in both automatic (ROUGE) and human evaluations, demonstrating superior ability to capture long-range dependencies and preserve key information through abstraction and fusion.

ABSTRACT

We introduce extreme summarization, a new single-document summarization task which does not favor extractive strategies and calls for an abstractive modeling approach. The idea is to create a short, one-sentence news summary answering the question "What is the article about?". We collect a real-world, large-scale dataset for this task by harvesting online articles from the British Broadcasting Corporation (BBC). We propose a novel abstractive model which is conditioned on the article's topics and based entirely on convolutional neural networks. We demonstrate experimentally that this architecture captures long-range dependencies in a document and recognizes pertinent content, outperforming an oracle extractive system and state-of-the-art abstractive approaches when evaluated automatically and by humans.

Motivation & Objective

• To define and formalize extreme summarization as a new single-document summarization task that resists extractive approaches and demands abstractive reasoning.
• To collect a large-scale, real-world dataset from BBC news articles, where the first sentence often serves as a one-sentence summary.
• To develop a novel abstractive model based entirely on convolutional neural networks (CNNs), conditioned on document topics, to better capture long-range dependencies and document-level abstraction.
• To demonstrate that topic-aware, CNN-based models significantly outperform both extractive and state-of-the-art RNN-based abstractive models in summarization quality.

Proposed method

• The proposed model, T-ConvS2S, uses a convolutional encoder to associate each word with a topic vector, capturing whether it is representative of the document’s content.
• The convolutional decoder generates each word in the summary conditioned on a global document topic vector, enabling context-aware abstractive generation.
• The model relies solely on convolutional layers, avoiding recurrent networks, to better capture long-range dependencies across the document.
• Topic vectors are learned end-to-end during training and used to guide both encoding and decoding, enhancing relevance and abstraction.
• The architecture is trained end-to-end using sequence-to-sequence learning with cross-entropy loss on the XSum dataset.
• The model is evaluated using automatic metrics (ROUGE) and two human evaluations: summary preference ranking and question-answering (QA) for key information retention.

Experimental results

Research questions

• RQ1 Can a purely convolutional neural network architecture effectively model long-range dependencies and document-level abstraction in extreme summarization?
• RQ2 Is an abstractive approach significantly superior to extractive methods for the extreme summarization task, where summaries must answer 'What is the article about?'?
• RQ3 Does conditioning the model on document topics improve the quality and informativeness of abstractive summaries compared to standard sequence-to-sequence models?
• RQ4 To what extent do human evaluators prefer summaries generated by the proposed topic-aware model over extractive or RNN-based abstractive systems?
• RQ5 How well do model-generated summaries preserve key factual information from the source document, as measured by question-answering performance?

Key findings

• The T-ConvS2S model achieved a ROUGE-L score of 46.05% on the XSum test set, significantly outperforming the extractive oracle (15.70%) and other abstractive models.
• In human preference evaluations, T-ConvS2S was ranked second, significantly preferred over ConvS2S and PtGen, and only behind human-written summaries.
• The QA evaluation showed that summaries from T-ConvS2S enabled participants to correctly answer 46.05% of fact-based questions, compared to 30.90% for ConvS2S and 21.40% for PtGen.
• The extractive oracle performed poorly in human evaluations (15.70% QA accuracy), indicating that ROUGE-optimized extractive summaries often fail to preserve key information.
• T-ConvS2S demonstrated superior ability to fuse and paraphrase information from scattered parts of the document, as evidenced by higher accuracy in answering questions requiring inference or synthesis.
• The model’s performance was statistically significant in both human evaluations, with T-ConvS2S significantly outperforming ConvS2S and PtGen (p < 0.01), confirming its robustness and effectiveness.