[Paper Review] Generating Query Focused Summaries from Query-Free Resources

Yumo Xu, Mirella Lapata|arXiv (Cornell University)|Dec 29, 2020

Topic Modeling41 references4 citations

TL;DR

This paper proposes Marge, a weakly supervised framework for abstractive query-focused summarization (QFS) that leverages generic summarization data without requiring query-related resources. By using a unified masked representation to convert summaries into proxy queries, Marge trains a query model for evidence ranking and generates coherent, query-controllable summaries, achieving state-of-the-art performance on multiple QFS benchmarks despite weak supervision.

ABSTRACT

The availability of large-scale datasets has driven the development of neural models that create generic summaries from single or multiple documents. In this work we consider query focused summarization (QFS), a task for which training data in the form of queries, documents, and summaries is not readily available. We propose to decompose QFS into (1) query modeling (i.e., finding supportive evidence within a set of documents for a query) and (2) conditional language modeling (i.e., summary generation). We introduce MaRGE, a Masked ROUGE Regression framework for evidence estimation and ranking which relies on a unified representation for summaries and queries, so that summaries in generic data can be converted into proxy queries for learning a query model. Experiments across QFS benchmarks and query types show that our model achieves state-of-the-art performance despite learning from weak supervision.

Motivation & Objective

• To address the lack of training data for query-focused summarization (QFS), which typically requires query-document-summary triples not readily available.
• To enable training of a query model without relying on question-answering or paraphrase datasets, reducing data dependency and cost.
• To develop a unified representation for summaries and queries to allow generic summarization data to be repurposed for QFS.
• To generate coherent, abstractive summaries that are both relevant to the query and concise, even under weak supervision.
• To achieve state-of-the-art performance in QFS using only generic summarization datasets, avoiding costly external query resources.

Proposed method

• Propose Marge, a Masked ROUGE Regression framework for evidence estimation and ranking, using a unified representation for summaries and queries.
• Convert summaries from generic summarization datasets into proxy queries via a masked representation, enabling weak supervision for query modeling.
• Decompose QFS into two stages: (1) query modeling via evidence ranking using Marge, and (2) conditional abstractive generation conditioned on selected evidence.
• Use a unified masked representation (UMR) to align summaries and queries in embedding space, allowing shared representation learning.
• Train the generator to produce abstractive summaries with controlled length and query relevance, using evidence from the ranked sentences.
• Leverage pre-trained models (e.g., BART) fine-tuned on generic summarization data for both the query model and the generator.

Experimental results

Research questions

• RQ1 Can generic summarization data be effectively repurposed to train a query-focused summarization system without access to query-related resources?
• RQ2 Can a unified representation for summaries and queries enable weak supervision for query modeling in QFS?
• RQ3 Does a two-stage framework—evidence ranking followed by abstractive generation—produce more coherent and relevant summaries than iterative or extractive baselines?
• RQ4 Can a model trained solely on generic summarization data achieve state-of-the-art performance in QFS across diverse domains and query types?
• RQ5 How does the coherence and relevance of generated summaries compare to those from models trained with distant supervision from QA datasets?

Key findings

• Marge achieves state-of-the-art performance on both evidence ranking and abstractive QFS across multiple benchmarks, including DUC and TD-QFS, despite using only generic summarization data.
• Human evaluation shows MargeSum-Cd is on par with PQSum-Wsl in relevance and succinctness, and significantly more coherent than PQSum-Wsl and QuerySum, which produce incoherent outputs due to sentence reordering.
• MargeSum-Cd outperforms UniLM-Cd in relevance (4.55 vs. 3.63 on TD-QFS) while maintaining high coherence (4.37 vs. 4.28), demonstrating the effectiveness of unified representation and end-to-end generation.
• The model achieves a human-rated relevance score of 2.91 on DUC and 4.55 on TD-QFS, significantly outperforming UniLM-Cd (3.63) and approaching gold standard relevance (4.70) on TD-QFS.
• The use of proxy queries derived from summaries enables effective weak supervision, allowing the model to learn query-relevant evidence ranking without access to QA or paraphrase datasets.
• The abstractive generation framework produces long, coherent summaries in a single pass, avoiding the coherence issues common in iterative, extractive-abstractive pipelines.