[Paper Review] Controlling the Amount of Verbatim Copying in Abstractive Summarization

Kaiqiang Song, Bingqing Wang+3 more|arXiv (Cornell University)|Nov 23, 2019

Topic Modeling45 references4 citations

TL;DR

This paper proposes a decoder-only Transformer model that controls verbatim copying in abstractive summarization by learning from single reference summaries. By separating prediction of seen (copyable) and unseen (generative) words during training and decoding, the model generates summaries across a spectrum from extractive to abstractive, achieving competitive ROUGE and BERTScore results while enabling flexible control over copying rates.

ABSTRACT

An abstract must not change the meaning of the original text. A single most effective way to achieve that is to increase the amount of copying while still allowing for text abstraction. Human editors can usually exercise control over copying, resulting in summaries that are more extractive than abstractive, or vice versa. However, it remains poorly understood whether modern neural abstractive summarizers can provide the same flexibility, i.e., learning from single reference summaries to generate multiple summary hypotheses with varying degrees of copying. In this paper, we present a neural summarization model that, by learning from single human abstracts, can produce a broad spectrum of summaries ranging from purely extractive to highly generative ones. We frame the task of summarization as language modeling and exploit alternative mechanisms to generate summary hypotheses. Our method allows for control over copying during both training and decoding stages of a neural summarization model. Through extensive experiments we illustrate the significance of our proposed method on controlling the amount of verbatim copying and achieve competitive results over strong baselines. Our analysis further reveals interesting and unobvious facts.

Motivation & Objective

• To enable neural abstractive summarizers to generate summaries with varying degrees of verbatim copying, despite being trained on only a single reference summary.
• To address the lack of control over copying in modern abstractive models, which typically produce fixed-copy-rate outputs.
• To develop a framework that allows control over copying during both training and decoding stages.
• To improve summary quality by reducing hallucination and preserving meaning through controlled copying.
• To demonstrate that a single-reference model can produce diverse summary hypotheses across the extractive-abstractive spectrum.

Proposed method

• The model frames summarization as a language modeling task using a decoder-only Transformer architecture with shared encoder-decoder parameters.
• It separates prediction into two components: words seen in the source text (copying) and words unseen (generation), enabling control over copying behavior.
• During training, the model is exposed to varying proportions of seen and unseen words to learn to favor or avoid copying.
• At decoding, search strategies (beam search, best-first search) and reranking methods (e.g., BP-Norm, SBWR) are used to encourage copying of source wording.
• The model uses warm-started pretrained representations for all parameters, including attention and copy mechanisms, improving training stability.
• Copy rate is defined as the percentage of n-grams in the summary that appear verbatim in the source text, serving as a quantitative control metric.

Experimental results

Research questions

• RQ1 Can a neural abstractive summarizer trained on a single reference summary generate summaries with varying degrees of verbatim copying?
• RQ2 How effective is the proposed method in controlling copying during both training and decoding stages?
• RQ3 Does the model achieve competitive automatic and human evaluation scores across extractive and abstractive summary types?
• RQ4 How do different decoding strategies (beam search vs. best-first) and reranking methods affect copying and summary quality?
• RQ5 Does the model generalize across domains, and how does performance degrade in cross-domain settings?

Key findings

• The model achieves competitive ROUGE and BERTScore results, outperforming strong baselines on the Gigaword dataset despite using only single-reference summaries.
• Beam search with SBWR reranking yields the highest performance, outperforming length normalization and best-first search.
• The model with higher copying (pure-extractive) degrades more gracefully in cross-domain settings than the more abstractive variant.
• Human evaluation shows both variants of the model outperform baselines in informativeness, grammaticality, and truthfulness, with the best-abstract variant scoring 2.865 on truthfulness.
• Best-worst scaling confirms that the model’s best-abstract variant is selected as the best summary 4.2% more often than the worst, indicating strong human preference.
• The method enables effective control over copying without requiring multiple reference summaries, demonstrating the feasibility of flexible summarization from single references.