[Paper Review] Multi-Document Summarization via Discriminative Summary Reranking

Xiaojun Wan, Ziqiang Cao+3 more|arXiv (Cornell University)|Jul 8, 2015

Topic Modeling30 references18 citations

TL;DR

This paper proposes a discriminative reranking approach for multi-document summarization that generates candidate summaries using an ILP framework and improves quality by ranking them with Ranking SVM. By leveraging diverse features across word, sentence, and summary levels, the method achieves robust and high-quality summaries across varied document sets, outperforming individual models on DUC benchmarks.

ABSTRACT

Existing multi-document summarization systems usually rely on a specific summarization model (i.e., a summarization method with a specific parameter setting) to extract summaries for different document sets with different topics. However, according to our quantitative analysis, none of the existing summarization models can always produce high-quality summaries for different document sets, and even a summarization model with good overall performance may produce low-quality summaries for some document sets. On the contrary, a baseline summarization model may produce high-quality summaries for some document sets. Based on the above observations, we treat the summaries produced by different summarization models as candidate summaries, and then explore discriminative reranking techniques to identify high-quality summaries from the candidates for difference document sets. We propose to extract a set of candidate summaries for each document set based on an ILP framework, and then leverage Ranking SVM for summary reranking. Various useful features have been developed for the reranking process, including word-level features, sentence-level features and summary-level features. Evaluation results on the benchmark DUC datasets validate the efficacy and robustness of our proposed approach.

Motivation & Objective

• To address the limitation of existing summarization models that perform inconsistently across different document sets.
• To improve summary quality by combining outputs from multiple summarization models as candidate summaries.
• To develop a robust reranking framework that selects the highest-quality summary for each document set.
• To explore feature engineering for discriminative reranking, including word-level, sentence-level, and summary-level features.

Proposed method

• Generate candidate summaries for each document set using an Integer Linear Programming (ILP) framework to optimize summary quality.
• Apply Ranking SVM to rerank candidate summaries based on learned relevance scores.
• Design a comprehensive set of features: word-level (e.g., term frequency, salience), sentence-level (e.g., position, length), and summary-level (e.g., coverage, diversity).
• Train the Ranking SVM model on annotated data to learn the ranking function that distinguishes high-quality from low-quality summaries.
• Use the reranked summary as the final output, ensuring topic coverage and coherence.
• Evaluate the framework on standard DUC benchmark datasets to validate performance and robustness.

Experimental results

Research questions

• RQ1 Can combining multiple summarization models improve summary quality compared to relying on a single model?
• RQ2 How effective is discriminative reranking in selecting the best summary from multiple candidate outputs?
• RQ3 Which features—word-level, sentence-level, or summary-level—are most predictive of summary quality?
• RQ4 Does the proposed method maintain high performance across diverse document sets with varying topics?
• RQ5 Can the reranking framework generalize well to unseen document sets?

Key findings

• The proposed method significantly improves summary quality compared to individual summarization models on the DUC benchmark datasets.
• Discriminative reranking with Ranking SVM achieves robust performance across diverse document sets, reducing variability in output quality.
• The inclusion of summary-level features such as coverage and diversity enhances the model’s ability to select comprehensive and non-redundant summaries.
• Word-level features like term frequency and salience contribute meaningfully to identifying important content.
• The method demonstrates consistent gains in ROUGE scores, indicating improved recall and F1 performance over baseline models.
• The framework is effective even when individual models produce low-quality summaries for certain document sets, highlighting its robustness.