[Paper Review] Efficient Adaptation of Pretrained Transformers for Abstractive Summarization

Andrew Hoang, Antoine Bosselut+2 more|arXiv (Cornell University)|Jun 1, 2019

Topic Modeling26 references39 citations

TL;DR

The paper adapts pretrained transformer language models for abstractive summarization using source embeddings and domain-adaptive training, achieving new state-of-the-art ROUGE on two datasets and strong human评价 performance.

ABSTRACT

Large-scale learning of transformer language models has yielded improvements on a variety of natural language understanding tasks. Whether they can be effectively adapted for summarization, however, has been less explored, as the learned representations are less seamlessly integrated into existing neural text production architectures. In this work, we propose two solutions for efficiently adapting pretrained transformer language models as text summarizers: source embeddings and domain-adaptive training. We test these solutions on three abstractive summarization datasets, achieving new state of the art performance on two of them. Finally, we show that these improvements are achieved by producing more focused summaries with fewer superfluous and that performance improvements are more pronounced on more abstractive datasets.

Motivation & Objective

• Motivate and assess how pretrained transformer representations can be effectively adapted for abstractive summarization.
• Propose two adaptation strategies—source embeddings and domain-adaptive training (DAT)—to bridge domain gaps and task alignment.
• Evaluate the approach on three major summarization datasets and compare with strong baselines.
• Analyze how summary length and extractiveness affect ROUGE scores and human judgments.

Proposed method

• Represent articles and summaries as a single input sequence with a delimiter and a special end token.
• Introduce source embeddings to distinguish article versus summary tokens in the input representation.
• Apply domain-adaptive training to align the model with newswire language before end-task summarization training.
• Fine-tune a GPT-based transformer as a summarizer with end-task training to maximize conditional likelihood of the target summary.
• Optionally ablate source embeddings to form Transformer-LM for comparison.
• Use beam search with a beam size of 3 for generation and evaluate with ROUGE metrics.

Experimental results

Research questions

• RQ1 Can pretrained transformer language models be effectively repurposed as abstractive summarizers when initialized from large-scale language models?
• RQ2 Do source embeddings and domain-adaptive training improve abstractive summarization quality across datasets?
• RQ3 How do ROUGE scores relate to human judgments and summary length in transformer-based summarizers?
• RQ4 Are current state-of-the-art summaries largely extractive despite abstractive design?

Key findings

• Transformer-based summarizers trained with source embeddings and domain-adaptive training achieve state-of-the-art ROUGE-L on two datasets.
• Human evaluation favored Transformer-SM over some strong baselines across non-redundancy, coherence, focus, and overall quality.
• Transformer-SM tends to produce shorter, more concise summaries yet maintains comparable ROUGE-L recall to longer baselines, indicating greater efficiency.
• ROUGE scores, especially ROUGE-L, are sensitive to summary length and can be optimistic when longer summaries are favored.
• On CNN/Daily Mail, Transformer-SM underperforms some baselines in ROUGE, but is preferred in human judgments for quality aspects.
• On XSum, Transformer-SM outperforms baselines and matches or exceeds Transformer-LM, suggesting benefits of both adaptations for highly abstractive summaries.
• On Newsroom, Transformer-SM surpasses prior best models across ROUGE-1, ROUGE-2, and ROUGE-L.