[Paper Review] Abstractive Document Summarization without Parallel Data

Nikola I. Nikolov, Richard H. R. Hahnloser|arXiv (Cornell University)|Jul 30, 2019

Topic Modeling28 references5 citations

TL;DR

This paper proposes an abstractive document summarization system that generates summaries without requiring parallel article-summary pairs, using only example summaries and non-matching articles. It combines an unsupervised sentence extractor with a sentence abstractor trained on pseudo-parallel and backtranslated synthetic data, achieving competitive performance on CNN/DailyMail and promising results on scientific press release generation—demonstrating viability in low-resource settings.

ABSTRACT

Abstractive summarization typically relies on large collections of paired articles and summaries. However, in many cases, parallel data is scarce and costly to obtain. We develop an abstractive summarization system that relies only on large collections of example summaries and non-matching articles. Our approach consists of an unsupervised sentence extractor that selects salient sentences to include in the final summary, as well as a sentence abstractor that is trained on pseudo-parallel and synthetic data, that paraphrases each of the extracted sentences. We perform an extensive evaluation of our method: on the CNN/DailyMail benchmark, on which we compare our approach to fully supervised baselines, as well as on the novel task of automatically generating a press release from a scientific journal article, which is well suited for our system. We show promising performance on both tasks, without relying on any article-summary pairs.

Motivation & Objective

• Address the challenge of abstractive summarization in low-resource settings where parallel article-summary pairs are scarce or unavailable.
• Develop a system that leverages only example summaries and non-matching articles, avoiding reliance on costly parallel corpora.
• Enable effective abstractive summarization for domains such as scientific publishing, where aligned data is limited.
• Demonstrate that unsupervised extraction combined with data-augmented abstraction can rival supervised baselines in performance.
• Explore a novel task: generating press releases from scientific journal articles without parallel training data.

Proposed method

• Use an unsupervised extractive model (e.g., Lead or LexRank) to identify salient sentences from input articles.
• Construct pseudo-parallel sentence pairs by aligning sentences from summaries and non-matching articles using large-scale embedding-based matching.
• Train a backtranslation model on the pseudo-parallel pairs to generate synthetic article sentences from summary sentences.
• Train a sentence abstractor (P_PM) on the combination of pseudo-parallel pairs and backtranslated synthetic data to paraphrase extracted sentences.
• Apply the abstractor to rephrase selected sentences into more fluent, compressed, and domain-adapted summaries.
• Use the final abstracted sentences as the output summary, achieving abstractive compression without reusing full source sentences.

Experimental results

Research questions

• RQ1 Can abstractive summarization be effectively trained without any parallel article-summary pairs?
• RQ2 How well does a system based on pseudo-parallel and synthetic data perform compared to fully supervised baselines on standard benchmarks?
• RQ3 Can such a method generalize to low-resource, domain-specific tasks like generating press releases from scientific articles?
• RQ4 To what extent can unsupervised sentence extraction combined with data-augmented abstraction produce fluent and informative summaries?
• RQ5 How does the performance gap between supervised and non-parallel methods compare across different summarization tasks?

Key findings

• On the CNN/DailyMail benchmark, the proposed method achieved competitive performance compared to a fully supervised LSTM-based abstractive model, despite not using any parallel training data.
• The system outperformed unsupervised extractive baselines (Lead and LexRank) on the scientific press release generation task, demonstrating the benefit of abstraction.
• The abstractor learned to compress and rephrase sentences effectively, often replacing technical phrasing with more accessible language suitable for press releases.
• The model trained on backtranslated synthetic data showed conservative paraphrasing, preserving most of the original sentence structure while still achieving meaningful compression.
• The gap between extractive baselines and the oracle (ideal extractive summary) was small, indicating that abstractive summarization is essential for high-quality results in low-resource settings.
• The final training dataset for the abstractor contained 8.6 million pseudo-parallel and synthetic sentence pairs, enabling effective pretraining without parallel supervision.