[Paper Review] QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization

Ming Zhong, Da Yin+9 more|arXiv (Cornell University)|Apr 13, 2021

Topic Modeling52 references27 citations

TL;DR

QMSum introduces a new benchmark for query-based meeting summarization across multiple domains and proposes a locate-then-summarize two-stage method with extensive baselines and multi-domain evaluation.

ABSTRACT

Meetings are a key component of human collaboration. As increasing numbers of meetings are recorded and transcribed, meeting summaries have become essential to remind those who may or may not have attended the meetings about the key decisions made and the tasks to be completed. However, it is hard to create a single short summary that covers all the content of a long meeting involving multiple people and topics. In order to satisfy the needs of different types of users, we define a new query-based multi-domain meeting summarization task, where models have to select and summarize relevant spans of meetings in response to a query, and we introduce QMSum, a new benchmark for this task. QMSum consists of 1,808 query-summary pairs over 232 meetings in multiple domains. Besides, we investigate a locate-then-summarize method and evaluate a set of strong summarization baselines on the task. Experimental results and manual analysis reveal that QMSum presents significant challenges in long meeting summarization for future research. Dataset is available at \url{https://github.com/Yale-LILY/QMSum}.

Motivation & Objective

• Define the task of query-based multi-domain meeting summarization and motivate its need for flexible, user-tailored meeting digests.
• Construct a high-quality, hierarchical-annotated dataset (QMSum) spanning multiple domains with queries, relevant spans, and summaries.
• Propose a locate-then-summarize pipeline and establish strong baselines to quantify challenges in long meeting summarization.

Proposed method

• Formalize the task as p(y|Q, X) where Q is a query and X is a meeting transcript.
• Implement a two-stage Locator that identifies query-relevant spans (using Pointer Network or hierarchical ranking), followed by a Summarizer that generates abstractive summaries from the extracted spans.
• Explore multiple summarizers (PGNet, BART, HMNet) with and without gold spans to assess the impact of the localization step.
• Construct and evaluate across three domains (Product, Academic, Committee) and analyze cross-domain generalization.
• Provide cross-domain pretraining and evaluation to test robustness in multi-domain settings.

Experimental results

Research questions

• RQ1 Can a locate-then-summarize approach effectively produce query-focused summaries for long, multi-domain meeting transcripts?
• RQ2 How does cross-domain training affect the generalizability of meeting summarization models across Product, Academic, and Committee domains?
• RQ3 What are the main challenges (e.g., factuality, relevance) in query-based meeting summarization and how do they vary by query type?

Key findings

• QMSum contains 1,808 query–summary pairs over 232 meetings across three domains.
• Locating relevant spans with the proposed Locator substantially improves input quality for the summarizer, achieving ROUGE-L recalls up to 84.04 when extracting 1/3 of the original text.
• Neural summarizers with Locator input outperform baselines, with HMNet achieving the best results among the tested models in the single-domain setting.
• Multi-domain training yields robust cross-domain performance and can outperform single-domain models on some domains (e.g., Academic: ROUGE-2 and ROUGE-L).
• Human evaluation reveals substantial factuality and relevance gaps in generated summaries, including 74% of samples with inconsistent facts and 31% unrelated to the query.