[Paper Review] PoBRL: Optimizing Multi-Document Summarization by Blending Reinforcement Learning Policies

Andy Su, Difei Su+2 more|arXiv (Cornell University)|May 18, 2021

Topic Modeling32 references4 citations

TL;DR

PoBRL proposes a novel reinforcement learning framework that decouples multi-document summarization into three objectives—importance, relevance, and length—by training separate policies via RL and blending them for improved summary quality. It achieves state-of-the-art ROUGE scores and superior human evaluation results on Multi-News and DUC-04 datasets, demonstrating reduced redundancy and enhanced coherence through policy blending.

ABSTRACT

We propose a novel reinforcement learning based framework PoBRL for solving multi-document summarization. PoBRL jointly optimizes over the following three objectives necessary for a high-quality summary: importance, relevance, and length. Our strategy decouples this multi-objective optimization into different subproblems that can be solved individually by reinforcement learning. Utilizing PoBRL, we then blend each learned policies together to produce a summary that is a concise and complete representation of the original input. Our empirical analysis shows state-of-the-art performance on several multi-document datasets. Human evaluation also shows that our method produces high-quality output.

Motivation & Objective

• To address the challenge of generating concise, non-redundant, and comprehensive summaries from multiple overlapping documents.
• To jointly optimize three key summarization objectives—importance, relevance (redundancy reduction), and length—through a decoupled reinforcement learning approach.
• To improve upon greedy or locally optimal methods like MMR by enabling global optimization via policy blending.
• To develop a scalable and effective framework that maintains high performance across varying numbers of input documents.

Proposed method

• Decomposes multi-document summarization into three independent sub-problems: sentence importance, relevance (redundancy minimization), and summary length control.
• Trains separate reinforcement learning policies for each objective using environment-specific reward functions.
• Employs a dynamic blending strategy that combines policies using a learnable λ coefficient derived from the advantage function.
• Uses MMR as a guiding mechanism to balance relevance and redundancy during policy training and blending.
• Leverages hierarchical document structure to improve policy generalization and sentence representation.
• Applies ROUGE-L and BERT-based cosine similarity to measure redundancy and evaluate summary quality.

Experimental results

Research questions

• RQ1 Can a multi-objective reinforcement learning framework jointly optimize importance, relevance, and length in multi-document summarization more effectively than single-policy approaches?
• RQ2 How does policy blending improve summary quality compared to using a single RL policy?
• RQ3 To what extent does the proposed method reduce redundancy while preserving salient information across diverse input document counts?
• RQ4 How does the adaptive λ coefficient based on advantage function compare to fixed λ values in policy blending?
• RQ5 Does the framework maintain consistent performance across varying numbers of input documents?

Key findings

• PoBRL achieves state-of-the-art ROUGE-L F1 score of 38.67 on the DUC-04 dataset, outperforming existing methods including RL w/o Blend (36.95) and OCCAMS_V (38.50).
• On the Multi-News dataset, PoBRL achieves a ROUGE-L F1 of 38.13 with λ=0.9 and 38.67 with adaptive λ, significantly outperforming baseline models.
• The model reduces redundancy by 91.6% in ROUGE-L overlap (0.19 vs. 2.28) and increases BERT-based cosine similarity (12.73 vs. 12.49), indicating lower sentence-level repetition.
• Human evaluation shows PoBRL achieves the highest average score of 4.68 in non-redundancy and 4.28 in grammar, outperforming CopyTransformer, Hi-Map, and MatchSum.
• The method maintains consistent performance across input documents ranging from 2 to 9, demonstrating robustness to input size variation.
• Policy blending significantly improves performance over single-policy RL (36.95 → 38.67 ROUGE-1), confirming the effectiveness of combining specialized policies.