[Paper Review] SKT5SciSumm -- Revisiting Extractive-Generative Approach for Multi-Document Scientific Summarization

Huy Quoc To, Liu, Ming+5 more|arXiv (Cornell University)|Feb 27, 2024

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper proposes SKT5SciSumm, a hybrid extractive-generative framework for multi-document scientific summarization that uses SPECTER embeddings with K-means clustering for extractive summarization and fine-tuned T5 models for abstractive generation. It achieves state-of-the-art performance on the Multi-XScience dataset, with ROUGE-L score of 19.88 and ROUGE-LSum of 33.23, outperforming prior methods.

ABSTRACT

Summarization for scientific text has shown significant benefits both for the research community and human society. Given the fact that the nature of scientific text is distinctive and the input of the multi-document summarization task is substantially long, the task requires sufficient embedding generation and text truncation without losing important information. To tackle these issues, in this paper, we propose SKT5SciSumm - a hybrid framework for multi-document scientific summarization (MDSS). We leverage the Sentence-Transformer version of Scientific Paper Embeddings using Citation-Informed Transformers (SPECTER) to encode and represent textual sentences, allowing for efficient extractive summarization using k-means clustering. We employ the T5 family of models to generate abstractive summaries using extracted sentences. SKT5SciSumm achieves state-of-the-art performance on the Multi-XScience dataset. Through extensive experiments and evaluation, we showcase the benefits of our model by using less complicated models to achieve remarkable results, thereby highlighting its potential in advancing the field of multi-document summarization for scientific text.

Motivation & Objective

• To address the challenge of summarizing long, complex scientific documents from multiple sources.
• To improve summarization quality by combining effective sentence extraction with high-quality abstractive generation.
• To demonstrate that simpler, modular components can outperform complex end-to-end models in scientific summarization.
• To evaluate the impact of different T5 model sizes on summarization performance for scientific text.

Proposed method

• Utilizes SPECTER, a citation-informed sentence transformer, to generate contextual embeddings for scientific sentences.
• Applies K-means clustering on SPECTER embeddings to extract the most salient sentences in an unsupervised manner.
• Fine-tunes T5 models (small, base, large, XL) on the extracted sentences to generate abstractive summaries.
• Employs a two-stage pipeline: first extract key sentences via clustering, then generate fluent, concise summaries via T5.
• Uses the Multi-XScience dataset for training, validation, and evaluation, focusing on generating 'related work' sections.
• Evaluates performance using standard ROUGE metrics (R-1, R-2, R-L, R-LSum) against strong baselines.

Experimental results

Research questions

• RQ1 Can a hybrid extractive-generative approach outperform existing state-of-the-art models in multi-document scientific summarization?
• RQ2 How does the choice of T5 model size affect summarization quality on scientific text?
• RQ3 Does using SPECTER embeddings with K-means clustering lead to better extractive performance than prior extractive methods?
• RQ4 Can a simpler, modular pipeline achieve superior results compared to more complex end-to-end models?

Key findings

• SKT5SciSumm achieves state-of-the-art performance on the Multi-XScience dataset, with ROUGE-1 of 37.49, ROUGE-2 of 8.23, ROUGE-L of 19.88, and ROUGE-LSum of 33.23.
• The T5-large model yields the highest ROUGE scores, outperforming smaller variants and the T5-XL model, which exhibited excessive paraphrasing of scientific phrases.
• The extractive component (SPECTER + K-means) achieves ROUGE-1 of 33.61, ROUGE-2 of 7.29, and ROUGE-LSum of 28.15, surpassing prior extractive methods by at least 1.46% in ROUGE-2.
• Compared to the previous SOTA model KGSum, SKT5SciSumm improves ROUGE-1 by 1.72%, ROUGE-2 by 0.74%, and ROUGE-LSum by 1.8%.
• The ablation study confirms that the SPECTER + K-means extractor significantly outperforms LexRank, TextRank, and LEAD baselines.
• Human evaluation suggests that T5-large generates summaries that are more fluent and structurally similar to human-written scientific summaries.