[Paper Review] On Word Error Rate Definitions and their Efficient Computation for Multi-Speaker Speech Recognition Systems

Thilo von Neumann, Christoph Boeddeker+3 more|arXiv (Cornell University)|Nov 29, 2022

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes a generalized MIMO Word Error Rate (WER) framework for multi-speaker, multi-channel ASR systems that efficiently computes WER using dynamic programming on a multi-dimensional Levenshtein distance tensor, ensuring utterances are consistently matched to a single hypothesis channel. The method enables polynomial-time computation of ORC WER—previously exponential—making it practical for long recordings and diverse evaluation scenarios.

ABSTRACT

We propose a general framework to compute the word error rate (WER) of ASR systems that process recordings containing multiple speakers at their input and that produce multiple output word sequences (MIMO). Such ASR systems are typically required, e.g., for meeting transcription. We provide an efficient implementation based on a dynamic programming search in a multi-dimensional Levenshtein distance tensor under the constraint that a reference utterance must be matched consistently with one hypothesis output. This also results in an efficient implementation of the ORC WER which previously suffered from exponential complexity. We give an overview of commonly used WER definitions for multi-speaker scenarios and show that they are specializations of the above MIMO WER tuned to particular application scenarios. We conclude with a discussion of the pros and cons of the various WER definitions and a recommendation when to use which.

Motivation & Objective

• Address the lack of a unified, efficient WER definition for multi-speaker, multi-channel ASR systems (MIMO) that process overlapping speech and produce multiple output sequences.
• Overcome the exponential computational complexity of existing WER definitions like ORC WER, which previously made evaluation infeasible for recordings with many utterances.
• Provide a generalization of existing WER definitions (e.g., uWER, SA-WER, cpWER, ORC WER) as special cases of a unified MIMO WER framework.
• Enable WER evaluation without requiring speaker labels or precise timing information, making it applicable to a wide range of ASR systems, including end-to-end models.
• Develop and release an open-source tool, MeetEval, for efficient MIMO WER computation with support for ORC WER and other variants.

Proposed method

• Formalize MIMO WER as a multi-dimensional Levenshtein distance computation over reference and hypothesis sequences, with the constraint that each utterance must be matched consistently to a single hypothesis channel.
• Introduce a channel change token in the dynamic programming formulation to penalize utterance splits across multiple output channels, ensuring coherent transcription evaluation.
• Derive an efficient dynamic programming algorithm with polynomial time complexity by structuring the search space as a tensor and applying constraints to avoid exponential enumeration.
• Optimize the ORC WER computation by reducing its complexity from exponential to polynomial in the number of utterances using the same framework, enabling evaluation on longer recordings.
• Implement the algorithm in the open-source MeetEval toolkit, supporting flexible input formats and integration with existing ASR evaluation pipelines.
• Benchmark the method against brute-force ORC WER, asclite, and other WER variants on real-world meeting recognition scenarios to validate efficiency and accuracy.

Experimental results

Research questions

• RQ1 How can a unified, general-purpose WER definition be formulated for MIMO ASR systems that process multiple speakers and produce multiple output channels?
• RQ2 What is the computational complexity of existing WER definitions like ORC WER, and can it be reduced from exponential to polynomial time?
• RQ3 How does the proposed MIMO WER framework relate to and generalize existing WER definitions such as uWER, SA-WER, cpWER, and ORC WER?
• RQ4 In what scenarios does the MIMO WER framework outperform or avoid the pitfalls of existing WER definitions, particularly in terms of robustness to utterance splitting and speaker misattribution?
• RQ5 Can the framework be efficiently implemented without relying on temporal or speaker-level annotations, and how does it compare to existing tools like asclite in terms of runtime and scalability?

Key findings

• The proposed MIMO WER framework generalizes existing WER definitions, with uWER, SA-WER, cpWER, and ORC WER as special cases under different constraints.
• The ORC WER computation complexity is reduced from exponential to polynomial time using dynamic programming on a multi-dimensional Levenshtein tensor, enabling evaluation on recordings with over 20 utterances—previously infeasible.
• The MIMO WER and ORC WER exhibit polynomial runtime complexity in the number of utterances, as confirmed by benchmarking on up to 100 utterances, with concave curves in log-scale runtime plots.
• The asclite tool shows near-linear performance in the benchmark due to temporal annotation usage, but its complexity explodes in complex scenarios (e.g., mms_msg), where only the proposed ORC WER implementation remains viable.
• The MIMO WER framework is independent of speaker labels and timing, making it applicable to end-to-end ASR systems and other architectures lacking such annotations.
• The open-source MeetEval toolkit enables efficient, scalable WER evaluation for MIMO systems and supports integration with existing evaluation pipelines, including for systems incompatible with ORC WER.