[Paper Review] AQuaMuSe: Automatically Generating Datasets for Query-Based Multi-Document Summarization

Sayali Kulkarni, Sheide Chammas+3 more|arXiv (Cornell University)|Oct 23, 2020

Topic Modeling34 references33 citations

TL;DR

AQuaMuSe automatically mines qMDS examples from question answering data and a large web corpus to create targeted multi-document summaries, yielding a large, configurable dataset for abstractive and extractive qMDS with 5,519 examples and an average of 6 source documents per query.

ABSTRACT

Summarization is the task of compressing source document(s) into coherent and succinct passages. This is a valuable tool to present users with concise and accurate sketch of the top ranked documents related to their queries. Query-based multi-document summarization (qMDS) addresses this pervasive need, but the research is severely limited due to lack of training and evaluation datasets as existing single-document and multi-document summarization datasets are inadequate in form and scale. We propose a scalable approach called AQuaMuSe to automatically mine qMDS examples from question answering datasets and large document corpora. Our approach is unique in the sense that it can general a dual dataset -- for extractive and abstractive summaries both. We publicly release a specific instance of an AQuaMuSe dataset with 5,519 query-based summaries, each associated with an average of 6 input documents selected from an index of 355M documents from Common Crawl. Extensive evaluation of the dataset along with baseline summarization model experiments are provided.

Motivation & Objective

• Motivate the need for large-scale, high-quality qMDS data for training and evaluation.
• Propose a scalable, automatic method to construct qMDS datasets from QA datasets and web corpora.
• Provide an instance of a dataset with both abstractive and extractive targets for qMDS.
• Demonstrate dataset quality through human evaluation and baseline model experiments.

Proposed method

• Encode sentences with Universal Sentence Encoder to measure semantic similarity between long-form answers and web passages.
• Select candidate input documents by nearest-neighbor matching of answer sentences against a large web corpus (Common Crawl) using similarity thresholds.
• Construct a top-K document set (K=7) with tunable thresholds to control diversity and topicality of inputs.
• Create abstractive qMDS examples by synthesizing a long-form answer a from matched passages R, enabling non-extractive generation.
• Create extractive qMDS examples by in-place substitution of matched sentences into the target answer.
• Evaluate dataset properties including coverage, density, summary recall (capped at 0.75), and document quality; provide baseline model experiments.

Experimental results

Research questions

• RQ1 How can we automatically generate large-scale qMDS datasets from QA data and web corpora?
• RQ2 Can such automatically generated data support both abstractive and extractive qMDS?
• RQ3 What are the quality characteristics of the resulting dataset (coverage, fluency, relevance) and how do baseline models perform on it?
• RQ4 How do dataset construction choices (thresholds, K) affect diversity and relevance of the inputs?

Key findings

• The authors generate 5,519 qMDS examples (train 4,555; dev 440; test 524) with an average of 6 input documents per example.
• The dataset uses a large Common Crawl base (355M pages) and a subset of Google Natural Questions as the target summaries.
• 85.18% of the abstractive examples are rated relevant by majority in a semantic correctness task.
• For fluency, 96.20% of edited extractive passages (with substituted matched sentences) were rated positive by raters; perplexity increased only slightly from 80 to 82.
• Summary recall is restricted to 0.75, ensuring a realistic challenge while maintaining reconstructability of the target summary.
• Baseline abstractive models (Hi-MAP, PEGASUS) and extractive baselines (NeuSum, TextRank) were evaluated under query-agnostic and query-based conditions to assess headroom and the impact of query conditioning.