[Paper Review] VCSUM: A Versatile Chinese Meeting Summarization Dataset

Han Wu, Mingjie Zhan+4 more|arXiv (Cornell University)|May 9, 2023

Topic Modeling4 citations

TL;DR

This paper introduces VCSum, a large-scale, versatile Chinese meeting summarization dataset comprising 239 real-life meetings totaling over 230 hours, with multi-granularity annotations including topic segmentation, headlines, segmentation summaries, overall summaries, and salient sentence highlights. The dataset enables diverse summarization tasks such as segmentation-based, multi-granularity, and retrieval-then-generate summarization, and establishes strong benchmark performance using pre-trained dialogue models like ConvLM.

ABSTRACT

Compared to news and chat summarization, the development of meeting summarization is hugely decelerated by the limited data. To this end, we introduce a versatile Chinese meeting summarization dataset, dubbed VCSum, consisting of 239 real-life meetings, with a total duration of over 230 hours. We claim our dataset is versatile because we provide the annotations of topic segmentation, headlines, segmentation summaries, overall meeting summaries, and salient sentences for each meeting transcript. As such, the dataset can adapt to various summarization tasks or methods, including segmentation-based summarization, multi-granularity summarization and retrieval-then-generate summarization. Our analysis confirms the effectiveness and robustness of VCSum. We also provide a set of benchmark models regarding different downstream summarization tasks on VCSum to facilitate further research. The dataset and code will be released at https://github.com/hahahawu/VCSum.

Motivation & Objective

• To address the scarcity of large-scale, high-quality Chinese meeting summarization datasets, which hinders progress in meeting summarization research.
• To create a versatile dataset that supports multiple summarization tasks, including segmentation-based, multi-granularity, and extract-then-generate summarization.
• To provide a comprehensive benchmark for evaluating summarization models on real-life Chinese meeting transcripts.
• To facilitate future research in end-to-end, multi-modal, and faithfulness-aware meeting summarization through a well-annotated, publicly released dataset.

Proposed method

• The dataset is constructed from publicly available, real-life Chinese meeting videos collected from Chinese video-sharing platforms, ensuring real-world relevance and diversity.
• Each meeting transcript is annotated with topic segmentation to identify thematic boundaries, enabling segmentation-based summarization.
• Multi-granularity summaries are provided: short headlines (5–20 words), detailed segmentation summaries (100–150 words), and overall meeting summaries (200–250 words).
• Salient sentences are explicitly highlighted by annotators to support extractive summarization and retrieval-then-generate methods.
• A dialogue-specific pre-trained language model, ConvLM, is fine-tuned on VCSum to establish baseline performance across multiple summarization tasks.
• Annotators are trained and monitored to ensure consistency, with sensitive information anonymized and ethical guidelines strictly followed.

Experimental results

Research questions

• RQ1 How effective is VCSum as a benchmark for segmentation-based meeting summarization?
• RQ2 Can multi-granularity summarization models achieve strong performance on real-life Chinese meeting data?
• RQ3 How well do retrieval-then-generate methods perform when using salient sentence highlights from VCSum?
• RQ4 What are the main error patterns in summarization models when applied to long, informal, multi-speaker meeting transcripts?
• RQ5 To what extent can dialogue-specific pre-training improve summarization performance on VCSum?

Key findings

• VCSum contains 239 real-life Chinese meetings with over 230 hours of total duration and an average of 14,000+ tokens per transcript, making it the largest such dataset in Chinese to date.
• The ConvLM model fine-tuned on VCSum achieves comparable or better performance than standard abstractive models on segmentation and joint summarization tasks, demonstrating the benefit of dialogue-specific pre-training.
• Approximately 50% of summarization errors are due to information missing from input truncation, especially in overall summary generation, highlighting the challenge of long-context modeling.
• About 20% of errors stem from irrelevant or redundant retrieved content in retrieval-then-generate methods, indicating a need for better retrieval quality.
• The remaining 30% of errors include factual and syntactic inaccuracies, suggesting ongoing challenges in generation faithfulness and coherence.
• The dataset is ethically constructed with anonymized data, fair compensation for annotators, and careful screening to avoid offensive or biased content.