[Paper Review] Multi-stage Pretraining for Abstractive Summarization

Sebastian Goodman, Zhenzhong Lan+1 more|arXiv (Cornell University)|Sep 23, 2019

Topic Modeling34 references4 citations

TL;DR

This paper proposes multi-stage pretraining for abstractive summarization, using full-network initialization with BERT and Gigaword pretraining to improve performance and abstraction. It achieves +1.05 ROUGE-L on Gigaword and +1.78 ROUGE-L on CNN/DailyMail over random initialization, with a 4% abstraction rate—significantly higher than prior MLE-based models—using only maximum-likelihood training and no inference-time heuristics.

ABSTRACT

Neural models for abstractive summarization tend to achieve the best performance in the presence of highly specialized, summarization specific modeling add-ons such as pointer-generator, coverage-modeling, and inferencetime heuristics. We show here that pretraining can complement such modeling advancements to yield improved results in both short-form and long-form abstractive summarization using two key concepts: full-network initialization and multi-stage pretraining. Our method allows the model to transitively benefit from multiple pretraining tasks, from generic language tasks to a specialized summarization task to an even more specialized one such as bullet-based summarization. Using this approach, we demonstrate improvements of 1.05 ROUGE-L points on the Gigaword benchmark and 1.78 ROUGE-L points on the CNN/DailyMail benchmark, compared to a randomly-initialized baseline.

Motivation & Objective

• To improve abstractive summarization performance without relying on complex inference-time heuristics or task-specific architectural modifications.
• To investigate whether multi-stage pretraining—starting from generic language modeling (BERT) and progressing to summarization-specific tasks (Gigaword)—can enhance model performance and abstraction.
• To evaluate the impact of full-network parameter initialization versus partial or shallow pretraining on summarization benchmarks.
• To determine whether pretraining can improve abstraction rates while maintaining high ROUGE scores, especially in comparison to state-of-the-art MLE-based models.

Proposed method

• The model uses a Transformer-based architecture with full-network initialization, where both encoder and decoder layers are initialized using pre-trained weights from BERT and the Gigaword pretraining stage.
• Multi-stage pretraining proceeds in two phases: first, the model is pre-trained on the BERT language modeling objective for general language understanding, then on the Gigaword summarization task for summarization-specific knowledge.
• The final fine-tuning stage uses maximum-likelihood estimation (MLE) loss on the target dataset (CNN/DailyMail or Gigaword), without any reinforcement learning or inference-time heuristics.
• The method compares different initialization schemes, including random initialization, BERT-only encoder initialization, and joint BERT and Gigaword pretraining for both encoder and decoder.
• The model is evaluated using ROUGE-L F1 and abstraction rate, with ablation studies on partial pretraining to assess the impact of layer initialization depth.
• The approach avoids pointer-generator or coverage mechanisms in the main experiments, focusing on pretraining as the primary performance booster.

Experimental results

Research questions

• RQ1 Does multi-stage pretraining—starting from BERT and progressing to Gigaword summarization—improve abstractive summarization performance compared to random initialization?
• RQ2 Can full-network parameter initialization, rather than shallow pretraining (e.g., only word embeddings), lead to better performance on summarization tasks?
• RQ3 To what extent does pretraining increase the abstraction rate of generated summaries, especially when compared to state-of-the-art MLE-based models?
• RQ4 Does the proposed method achieve high performance without relying on inference-time heuristics such as length penalties or n-gram repetition control?
• RQ5 What is the relationship between the number of pre-trained layers and model performance, and does deeper pretraining consistently improve results?

Key findings

• The multi-stage pretraining approach achieved a +1.05 ROUGE-L F1 improvement on the Gigaword benchmark compared to a randomly-initialized baseline.
• On the CNN/DailyMail benchmark, the method achieved a +1.78 ROUGE-L F1 improvement over the same baseline, outperforming previous MLE-based models.
• The model reached a 4% abstraction rate, significantly higher than the 0.5% observed in the state-of-the-art MLE-based model by Gehrmann et al. (2018), indicating better balance between extractive and abstractive generation.
• Full-network initialization—where both encoder and decoder are initialized with pre-trained weights—yielded better performance than partial or shallow pretraining, with a strong positive correlation (r = 0.8698) between the number of pre-trained layers and ROUGE-L F1 score.
• The model achieved peak performance in fewer training epochs than zero- or one-stage pretraining baselines, indicating faster convergence.
• The addition of content-selection prediction yielded only a 0.03 ROUGE-L F1 increase, suggesting limited benefit compared to the gains from pretraining alone.