[Paper Review] Abstractive Summarization with Combination of Pre-trained Sequence-to-Sequence and Saliency Models

Itsumi Saito, Kyosuke Nishida+2 more|arXiv (Cornell University)|Mar 29, 2020

Topic Modeling23 references30 citations

TL;DR

The paper investigates nine combinations of pre-trained seq-to-seq models with token/ sentence-level saliency models, introducing a new CIT model that feeds important tokens as input, and shows improvements on CNN/DM and XSum datasets, notably surpassing fine-tuned baselines on CNN/DM.

ABSTRACT

Pre-trained sequence-to-sequence (seq-to-seq) models have significantly improved the accuracy of several language generation tasks, including abstractive summarization. Although the fluency of abstractive summarization has been greatly improved by fine-tuning these models, it is not clear whether they can also identify the important parts of the source text to be included in the summary. In this study, we investigated the effectiveness of combining saliency models that identify the important parts of the source text with the pre-trained seq-to-seq models through extensive experiments. We also proposed a new combination model consisting of a saliency model that extracts a token sequence from a source text and a seq-to-seq model that takes the sequence as an additional input text. Experimental results showed that most of the combination models outperformed a simple fine-tuned seq-to-seq model on both the CNN/DM and XSum datasets even if the seq-to-seq model is pre-trained on large-scale corpora. Moreover, for the CNN/DM dataset, the proposed combination model exceeded the previous best-performed model by 1.33 points on ROUGE-L.

Motivation & Objective

• Assess whether saliency models can complement pre-trained seq-to-seq models for abstractive summarization.
• Evaluate nine combinations of saliency and pre-trained models on standard datasets.
• Propose a new CIT model that explicitly includes important tokens as input to the summarization model.

Proposed method

• Use a Transformer-based encoder-decoder pre-trained on large unlabeled data (e.g., BART LARGE) as the seq-to-seq backbone.
• Develop two saliency model types: a shared encoder that jointly trains with the seq-to-seq model, and an extractor that selects important tokens/sentences.
• Train saliency models with pseudo reference labels derived from alignments between source and reference summaries; use binary cross-entropy loss for saliency.
• Define multiple combination architectures: Multi-Task (MT), Selective Encoding (SE), Selective Attention (SA), and their combinations with extractors (SEG, CIT, CIT+SE, CIT+SA).
• Propose CIT: feed K top saliency tokens C as additional input to the seq-to-seq model alongside the source text.

Experimental results

Research questions

• RQ1 Do saliency models improve abstractive summarization when used with pre-trained seq-to-seq models on extractive vs. abstractive datasets?
• RQ2 Which combination of saliency strategy (shared encoder vs extractor) yields the best ROUGE scores on CNN/DM and XSum?
• RQ3 Does the proposed CIT model outperform other fine-tuned and combination models without additional pre-training data?
• RQ4 How does the quality of pseudo saliency labels affect performance on CNN/DM versus XSum?

Key findings

• All combination models outperformed the simple fine-tuned BART on CNN/DM, with CIT and CIT+SE achieving the highest ROUGE scores.
• On CNN/DM, CIT achieved R1=45.74, R2=22.50, RL=42.44, outperforming the fine-tuned baseline.
• On CNN/DM, CIT+SE achieved R1=45.80, R2=22.53, RL=42.48, the best among reported methods for CNN/DM.
• On XSum, CIT performed best among combinations with R1=46.72, R2=20.53, RL=37.73 for “Top-K tokens” extractor configuration; overall improvements were smaller than CNN/DM.
• CIT outperformed other combination models without requiring extra pre-training data, indicating saliency-guided input can enhance pre-trained seq-to-seq summarization, especially for extractive datasets.
• Compared to state-of-the-art pre-trained models, CIT surpassed many on CNN/DM but PEGASUS HugeNews achieved top scores on XSum, highlighting dataset-dependent benefits.