[Paper Review] Pre-training for Abstractive Document Summarization by Reinstating Source Text

Yanyan Zou, Xingxing Zhang+3 more|arXiv (Cornell University)|Apr 4, 2020

Topic Modeling47 references4 citations

TL;DR

This paper proposes three sequence-to-sequence pre-training objectives—sentence reordering, next sentence generation, and masked document generation—that enable pre-training abstractive summarization models on unlabeled text by reconstructing the original source document. Despite using only 19GB of pre-training data, the method achieves ROUGE scores comparable to or better than models pre-trained on 160GB+ corpora, demonstrating strong effectiveness with limited data.

ABSTRACT

Abstractive document summarization is usually modeled as a sequence-to-sequence (Seq2Seq) learning problem. Unfortunately, training large Seq2Seq based summarization models on limited supervised summarization data is challenging. This paper presents three pre-training objectives which allow us to pre-train a Seq2Seq based abstractive summarization model on unlabeled text. The main idea is that, given an input text artificially constructed from a document, a model is pre-trained to reinstate the original document. These objectives include sentence reordering, next sentence generation, and masked document generation, which have close relations with the abstractive document summarization task. Experiments on two benchmark summarization datasets (i.e., CNN/DailyMail and New York Times) show that all three objectives can improve performance upon baselines. Compared to models pre-trained on large-scale data (more than 160GB), our method, with only 19GB text for pre-training, achieves comparable results, which demonstrates its effectiveness.

Motivation & Objective

• To address the challenge of training large abstractive summarization models with limited supervised data by leveraging unsupervised pre-training.
• To design pre-training objectives that explicitly model content reordering, a key characteristic of abstractive summaries.
• To improve model generalization and performance on abstractive summarization by pre-training on unlabeled text using sequence-to-sequence objectives.
• To demonstrate that effective pre-training for abstractive summarization is possible with significantly less data than current state-of-the-art models.

Proposed method

• Proposes three sequence-to-sequence pre-training (STEP) objectives: sentence reordering (SR), next sentence generation (NSG), and masked document generation (MDG).
• Each objective trains a seq2seq model to reconstruct the original source document from artificially perturbed versions—e.g., shuffled sentences, next-segment prediction, or masked spans.
• The model is pre-trained on unlabeled documents from the training split of summarization datasets (e.g., CNN/DailyMail, NYT) before fine-tuning on supervised summarization tasks.
• Uses a Transformer-based seq2seq architecture with shared encoder-decoder structure, pre-trained using masked language modeling and denoising objectives.
• Employs a two-stage training pipeline: pre-training on unlabeled data using STEP objectives, followed by fine-tuning on supervised summarization data.
• Evaluates the effectiveness of individual and combined STEP objectives on benchmark datasets using ROUGE and human evaluation.

Experimental results

Research questions

• RQ1 Can pre-training on unlabeled text using source text reinstatement improve abstractive summarization performance?
• RQ2 Do sequence-to-sequence pre-training objectives that model content reordering enhance summarization quality?
• RQ3 Can a model pre-trained on only 19GB of text achieve performance comparable to models pre-trained on 160GB or more?
• RQ4 How does the proposed STEP method compare to strong baselines like BERTAbs and UniLM in both automatic and human evaluation?
• RQ5 Does pre-training on the training split of a summarization dataset still yield generalization benefits?

Key findings

• The proposed STEP pre-training method, using only 19GB of unlabeled text, achieves ROUGE-1, ROUGE-2, and ROUGE-L scores on CNN/DailyMail that are comparable to or higher than models pre-trained on 160GB or more of data.
• The sentence reordering (SR) objective alone improves performance over strong baselines, even when fine-tuned on the same supervised data.
• Human evaluation shows that summaries generated by the best STEP model are ranked as the best in 23% of cases and have a significantly lower mean rank (2.77) than all other models except human references.
• The STEP model outperforms both ProphetNet (16GB) and UniLM, which are pre-trained on 16GB of text, demonstrating superior data efficiency.
• The model achieves higher ROUGE-1 and ROUGE-L scores than T5 and PEGASUS (HugeNews), though it lags slightly in ROUGE-2, indicating strong performance in capturing overall content and fluency.
• Qualitative analysis confirms that the model generates fluent, informative, and succinct summaries that often rephrase and reorder content from the source, reflecting abstractive behavior.