[Paper Review] Combining State-of-the-Art Models with Maximal Marginal Relevance for Few-Shot and Zero-Shot Multi-Document Summarization

David Adams, Gandharv Suri+1 more|arXiv (Cornell University)|Nov 19, 2022

Topic Modeling4 citations

TL;DR

This paper proposes a novel MMR-based framework that combines outputs from state-of-the-art pretrained models to improve few-shot and zero-shot multi-document summarization (MDS). By prioritizing query relevance over diversity in the MMR scoring, the method enhances single-term similarity to human summaries while maintaining state-of-the-art performance across ROUGE metrics, particularly excelling in low-resource settings.

ABSTRACT

In Natural Language Processing, multi-document summarization (MDS) poses many challenges to researchers above those posed by single-document summarization (SDS). These challenges include the increased search space and greater potential for the inclusion of redundant information. While advancements in deep learning approaches have led to the development of several advanced language models capable of summarization, the variety of training data specific to the problem of MDS remains relatively limited. Therefore, MDS approaches which require little to no pretraining, known as few-shot or zero-shot applications, respectively, could be beneficial additions to the current set of tools available in summarization. To explore one possible approach, we devise a strategy for combining state-of-the-art models' outputs using maximal marginal relevance (MMR) with a focus on query relevance rather than document diversity. Our MMR-based approach shows improvement over some aspects of the current state-of-the-art results in both few-shot and zero-shot MDS applications while maintaining a state-of-the-art standard of output by all available metrics.

Motivation & Objective

• To address the challenge of limited MDS-specific training data by enabling effective few-shot and zero-shot summarization.
• To improve summary quality in low-resource settings where fine-tuning is impractical or too costly.
• To combine diverse outputs from multiple pretrained models while minimizing redundancy and maximizing relevance to the query.
• To demonstrate that MMR-based fusion can achieve state-of-the-art performance without requiring model fine-tuning on MDS data.
• To explore whether query-focused MMR weighting can outperform diversity-focused approaches in MDS.

Proposed method

• The framework aggregates summaries from multiple pretrained models, including both MDS- and SDS-fine-tuned models.
• Each document cluster is processed as a single document for models pretrained on MDS, and split into individual documents for others.
• The MMR algorithm is applied to the combined summary outputs, with a strong bias toward query relevance rather than inter-summary diversity.
• Relevance is computed using semantic similarity between candidate sentences and the LDA-generated query document.
• The final summary is constructed by iteratively selecting sentences that maximize MMR score, balancing relevance and redundancy.
• The method uses ROUGE metrics to evaluate performance on Multi-News and WCEP datasets under both few-shot and zero-shot settings.

Experimental results

Research questions

• RQ1 Can combining multiple state-of-the-art models via MMR improve MDS performance in few-shot and zero-shot settings?
• RQ2 Does prioritizing query relevance over diversity in MMR selection lead to better ROUGE scores and summary quality?
• RQ3 How does the proposed method compare to individual models and existing SOTA approaches in low-resource MDS scenarios?
• RQ4 To what extent does the inclusion of non-MDS-pretrained models enhance performance, especially in zero-shot applications?
• RQ5 Can MMR-based fusion maintain readability and coherence while improving term-level relevance?

Key findings

• The MMR-combined approach achieved the highest ROUGE-1 score (46.23) on the Multi-News dataset, outperforming all individual models.
• On the WCEP dataset, the method achieved a ROUGE-1 score of 30.74, showing a significant improvement over all baseline models in the zero-shot setting.
• Despite lower ROUGE-2 and ROUGE-L scores than the PEGASUS-MDS model, the MMR-combination method maintained state-of-the-art performance across all metrics.
• The relative improvement over SOTA models was larger in the zero-shot WCEP experiment, indicating stronger effectiveness in low-resource scenarios.
• The method preserved readability and coherence by selecting full sentences, avoiding fragmented or incoherent outputs.
• The results suggest that non-MDS-pretrained models contribute meaningfully to performance, especially when fine-tuning is not feasible.