[Paper Review] Combination of abstractive and extractive approaches for summarization of long scientific texts

Vladislav Tretyak, Denis Stepanov|arXiv (Cornell University)|Jun 9, 2020

Topic Modeling11 references4 citations

TL;DR

This paper proposes a hybrid summarization method that combines extractive and abstractive transformer-based models to generate high-quality summaries of long scientific texts. It first uses a pre-trained BERT model to extract key sentences, then conditions a BART abstractive model on the extractive summary, introduction, and conclusion—achieving state-of-the-art ROUGE scores of 45.3 (Rouge-1), 25.1 (Rouge-2), and 36.2 (Rouge-L) on the arXiv dataset.

ABSTRACT

In this research work, we present a method to generate summaries of long scientific documents that uses the advantages of both extractive and abstractive approaches. Before producing a summary in an abstractive manner, we perform the extractive step, which then is used for conditioning the abstractor module. We used pre-trained transformer-based language models, for both extractor and abstractor. Our experiments showed that using extractive and abstractive models jointly significantly improves summarization results and ROUGE scores.

Motivation & Objective

• To improve summarization quality for long scientific documents by combining extractive and abstractive approaches.
• To investigate the impact of different conditioning inputs (e.g., extractive summary, introduction, conclusion) on abstractive summarization performance.
• To evaluate whether pre-trained transformer models can outperform prior methods on long-form scientific text summarization.
• To determine the optimal combination of extractive and abstractive components for improved ROUGE scores.
• To validate the hypothesis that extractive summaries and key sections (intro/conclusion) contain the most relevant information for abstractive generation.

Proposed method

• An extractive summarization model based on BERT, RoBERTa, or ELECTRA is trained as a sentence classification task to identify salient sentences.
• The extracted summary is concatenated with the introduction and conclusion of the paper to form a conditioning input for the abstractive model.
• A BART-based abstractive model is conditioned on the combined input (extracted summary + intro + conclusion) to generate natural language summaries.
• Alternative conditioning configurations are tested, including conditioning only on the extractive summary, introduction, or introduction and conclusion.
• GPT-2 is used for abstractive generation with conditioning on extractive summaries, though input length limits restrict full-length conditioning.
• The model is fine-tuned end-to-end with cross-entropy loss, and performance is evaluated using ROUGE metrics on the arXiv dataset.

Experimental results

Research questions

• RQ1 Does combining extractive and abstractive models improve summarization quality on long scientific texts?
• RQ2 Which conditioning input (extractive summary, introduction, conclusion, or combinations) yields the best abstractive summarization performance?
• RQ3 How does using pre-trained models (BERT, BART) compare to training from scratch in this hybrid setup?
• RQ4 Can abstractive generation with conditioning produce ROUGE scores exceeding the oracle extractive baseline?
• RQ5 Does paraphrasing the extractive summary degrade performance due to loss of alignment with ground truth?

Key findings

• The best-performing model, using BERT for extraction and BART for abstractive generation with conditioning on extractive summary, introduction, and conclusion, achieved ROUGE-1: 45.3, ROUGE-2: 25.1, and ROUGE-L: 36.2.
• Removing the extractive summary from the conditioning input led to a significant drop in ROUGE scores, confirming its critical role in guiding abstractive generation.
• Conditioning on the extractive summary combined with introduction and conclusion outperformed all other configurations, including using only introduction or conclusion.
• The model's ROUGE-L score of 36.2 surpassed the oracle extractive baseline (36.0), indicating that abstractive generation can produce more concise, relevant summaries than extractive methods alone.
• Paraphrasing the extractive summary for conditioning severely degraded performance (ROUGE-1: 17.9, ROUGE-2: 3.5), likely due to misalignment with ground truth.
• The BART model conditioned on extractive summary + intro + conclusion outperformed previous state-of-the-art approaches on the arXiv dataset across all ROUGE metrics.