[Paper Review] MeetEval: A Toolkit for Computation of Word Error Rates for Meeting Transcription Systems

Thilo von Neumann, Christoph Boeddeker+2 more|arXiv (Cornell University)|Jul 21, 2023

Natural Language Processing TechniquesComputer Science3 citations

TL;DR

MeetEval is an open-source toolkit that implements and extends Word Error Rate (WER) metrics for meeting transcription systems, introducing a time-constrained WER (tcpWER) that improves matching accuracy by enforcing plausible temporal alignments. It achieves better error assessment and faster computation by approximating word-level timings from segment-level annotations, with a 5-second collar yielding robust, realistic WER scores across diverse datasets.

ABSTRACT

MeetEval is an open-source toolkit to evaluate all kinds of meeting transcription systems. It provides a unified interface for the computation of commonly used Word Error Rates (WERs), specifically cpWER, ORC-WER and MIMO-WER along other WER definitions. We extend the cpWER computation by a temporal constraint to ensure that only words are identified as correct when the temporal alignment is plausible. This leads to a better quality of the matching of the hypothesis string to the reference string that more closely resembles the actual transcription quality, and a system is penalized if it provides poor time annotations. Since word-level timing information is often not available, we present a way to approximate exact word-level timings from segment-level timings (e.g., a sentence) and show that the approximation leads to a similar WER as a matching with exact word-level annotations. At the same time, the time constraint leads to a speedup of the matching algorithm, which outweighs the additional overhead caused by processing the time stamps.

Motivation & Objective

• To provide a unified, accessible toolkit for computing multiple WER metrics used in meeting transcription evaluation.
• To address the lack of standardized, efficient, and extensible tools for modern meeting transcription systems, especially those with multi-stream or overlapping speech outputs.
• To improve WER reliability by incorporating temporal constraints into matching, penalizing systems with poor time annotations.
• To enable accurate WER computation using coarse segment-level timings when exact word-level timings are unavailable.
• To optimize performance by reducing the search space in matching algorithms through temporal pruning.

Proposed method

• Introduces tcpWER, an extension of cpWER that enforces temporal plausibility by requiring matched words to fall within a user-defined collar around the reference time.
• Proposes a word-length-based heuristic to approximate word-level timings from segment-level timestamps, enabling WER computation without requiring high-precision word-level annotations.
• Employs dynamic programming with pruning to accelerate matching by excluding temporally implausible word pairs, reducing runtime complexity.
• Uses a collar (e.g., ±5 seconds) around reference word times to define acceptable temporal alignment windows, improving matching realism.
• Supports multiple output formats (diarization-style, CSS-style, SOT-style) and integrates with common system outputs, ensuring broad compatibility.
• Provides a unified API for cpWER, ORC-WER, MIMO-WER, and tcpWER, enabling easy metric switching and reproducible evaluation.

Experimental results

Research questions

• RQ1 How can WER metrics be improved to reflect actual transcription quality by incorporating temporal constraints?
• RQ2 Can word-level timings be reliably approximated from segment-level annotations without significant loss in WER accuracy?
• RQ3 What is the impact of temporal constraints on WER scores and computational efficiency in multi-speaker meeting transcription?
• RQ4 How does the choice of collar size affect the robustness and realism of WER scores across diverse datasets?
• RQ5 Can temporal pruning significantly reduce the computational cost of WER computation without compromising matching quality?

Key findings

• The tcpWER metric consistently produces higher WER values than cpWER, with larger differences observed on datasets with poorer recognition performance (e.g., CHiME-6: 62.4% vs. 66.6%).
• Using a 5-second collar leads to realistic and robust WER scores, as it balances the need to penalize unrealistic alignments while allowing for natural speech pauses.
• The approximation of word-level timings from segment-level annotations yields WER values nearly identical to those obtained with exact word-level timings, validating the heuristic approach.
• The temporal constraint in tcpWER reduces the search space, resulting in a measurable speedup—execution time decreases with longer recordings due to effective pruning.
• The toolkit achieves acceptable runtime performance, with execution times remaining practical even for long-form recordings (e.g., up to 134 minutes), and the speedup becomes more pronounced as recording length increases.
• The method effectively reduces the number of unrealistic word matchings, especially in scenarios with long speaker inactivity or high speaker counts, which otherwise inflate WER differences between cpWER and tcpWER.