[Paper Review] Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval

John Giorgi, Luca Soldaini+5 more|arXiv (Cornell University)|Dec 20, 2022

Topic Modeling4 citations

TL;DR

This paper investigates open-domain multi-document summarization (MDS), where summarizers must generate summaries from retrieved documents rather than gold-standard document sets. It finds that state-of-the-art summarizers suffer significant performance drops under imperfect retrieval, especially due to irrelevant documents, but performance improves with fine-tuning on open-domain data. The study provides practical guidelines for retrieval length and error robustness in real-world MDS applications.

ABSTRACT

Multi-document summarization (MDS) assumes a set of topic-related documents are provided as input. In practice, this document set is not always available; it would need to be retrieved given an information need, i.e. a question or topic statement, a setting we dub "open-domain" MDS. We study this more challenging setting by formalizing the task and bootstrapping it using existing datasets, retrievers and summarizers. Via extensive automatic and human evaluation, we determine: (1) state-of-the-art summarizers suffer large reductions in performance when applied to open-domain MDS, (2) additional training in the open-domain setting can reduce this sensitivity to imperfect retrieval, and (3) summarizers are insensitive to the retrieval of duplicate documents and the order of retrieved documents, but highly sensitive to other errors, like the retrieval of irrelevant documents. Based on our results, we provide practical guidelines to enable future work on open-domain MDS, e.g. how to choose the number of retrieved documents to summarize. Our results suggest that new retrieval and summarization methods and annotated resources for training and evaluation are necessary for further progress in the open-domain setting.

Motivation & Objective

• To formalize and study open-domain multi-document summarization (MDS), where document retrieval is required before summarization.
• To evaluate the performance degradation of state-of-the-art summarizers when applied to retrieved, rather than gold-standard, document sets.
• To investigate which types of retrieval errors most severely impact summarization quality.
• To assess whether fine-tuning on open-domain data reduces sensitivity to retrieval errors.
• To provide practical, data-driven guidelines for future research and deployment in open-domain MDS.

Proposed method

• Formalized open-domain MDS as a retrieve-then-summarize pipeline using queries and large document indexes.
• Bootstrapped the task using existing datasets (e.g., Multi-News, Cochrane, WCEP-10), pre-trained retrievers (e.g., BM25, DPR), and summarizers (e.g., PEGASUS, PRIMERA, LSG-BART).
• Applied controlled perturbations to input document sets—such as random replacement, deletion, duplication, and back-translation—to simulate retrieval errors.
• Conducted both automatic evaluation (ROUGE, BERTScore) and human evaluation with coverage and informativeness metrics.
• Fine-tuned summarizers on open-domain data to assess robustness improvements under retrieval noise.
• Used binomial tests and inter-annotator agreement (kappa > 0.21) to validate human evaluation results.

Experimental results

Research questions

• RQ1 How does performance of state-of-the-art summarizers degrade when applied to retrieved document sets instead of gold-standard document sets?
• RQ2 Which types of retrieval errors (e.g., irrelevant, duplicate, reordered documents) most significantly reduce summarization quality?
• RQ3 Can fine-tuning summarizers on open-domain data reduce their sensitivity to imperfect retrieval?
• RQ4 How does the number of retrieved documents affect summarization quality and robustness?
• RQ5 Are summarizers sensitive to document order or duplication, or do they primarily fail due to irrelevance?

Key findings

• State-of-the-art summarizers suffer large performance drops in open-domain MDS, with ROUGE-F1 decreasing by up to 20% on average under imperfect retrieval.
• Summarizers are highly sensitive to the retrieval of irrelevant documents, which cause the most significant performance degradation.
• Summarizers show low sensitivity to document duplication and order changes, indicating robustness to these types of errors.
• Fine-tuning summarizers on open-domain data significantly reduces their sensitivity to retrieval errors, improving robustness.
• Human evaluation confirms a statistically significant preference for gold-standard inputs over retrieved ones, with fair inter-annotator agreement (kappa > 0.21).
• The study identifies that 60–80% of retrieved documents must be relevant to maintain acceptable summarization quality, suggesting optimal retrieval length is critical.