[Paper Review] Prioritizing Speech Test Cases

Zhou Yang, Jieke Shi+5 more|arXiv (Cornell University)|Feb 1, 2023

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes Prophet, a tool that prioritizes speech test cases for automated speech recognition (ASR) systems by predicting which words in a reference text are likely to be misrecognized, using pre-trained language models to compute error scores without executing the ASR system. Evaluated across 3 ASR systems and 12 datasets, Prophet uncovers 12.63% more wrongly recognized words than the state-of-the-art method under the same testing budget.

ABSTRACT

With the wide adoption of automated speech recognition (ASR) systems, it is increasingly important to test and improve ASR systems. However, collecting and executing speech test cases is usually expensive and time-consuming, motivating us to strategically prioritize speech test cases. A key question is: how to determine the ideal order of collecting and executing speech test cases to uncover more errors as early as possible? Each speech test case consists of a piece of audio and the corresponding reference text. In this work, we propose PROPHET (PRiOritizing sPeecH tEsT), a tool that predicts potential error-uncovering speech test cases only based on their reference texts. Thus, PROPHET analyzes test cases and prioritizes them without running the ASR system, which can analyze speech test cases at a large scale. We evaluate 6 different prioritization methods on 3 ASR systems and 12 datasets. Given the same testing budget, we find that our approach uncovers 12.63% more wrongly recognized words than the state-of-the-art method. We select test cases from the prioritized list to fine-tune ASR systems and analyze how our approach can improve the ASR system performance. Statistical tests show that our proposed method can bring significantly larger performance improvement to ASR systems than the existing baseline methods. Furthermore, we perform correlation analysis and confirm that fine-tuning an ASR system using a dataset, on which the model performs worse, tends to improve the performance more.

Motivation & Objective

• To address the high cost and time consumption of collecting and executing speech test cases in ASR system testing.
• To identify a strategy for prioritizing test cases that maximizes early error detection with limited resources.
• To develop a method that predicts error-prone words in reference texts without requiring audio or ASR execution.
• To evaluate whether prioritizing test cases based on predicted error likelihood improves ASR model fine-tuning.
• To investigate which features of a test case suite correlate most strongly with improved ASR performance.

Proposed method

• Prophet uses pre-trained language models (e.g., BERT, RoBERTa) to analyze reference texts at the word level and predict which words are likely to be misrecognized by an ASR system.
• It computes a comprehensive error score per test case based on the predicted difficulty of individual words, reflecting their potential to uncover errors.
• The error score integrates linguistic features such as word rarity, phonetic complexity, and contextual ambiguity derived from contextual embeddings.
• Test cases are ranked by their overall error score, enabling prioritization for collection or execution without running the ASR model.
• The method is evaluated using 3 ASR systems and 12 diverse datasets, comparing prioritization strategies under fixed testing budgets.
• Fine-tuning experiments are conducted using test cases selected by Prophet and baselines to assess model improvement.

Experimental results

Research questions

• RQ1 Can we predict which words in a reference text are likely to be misrecognized by an ASR system using only the text, without audio or model inference?
• RQ2 Does prioritizing test cases based on predicted error likelihood lead to earlier and more comprehensive error detection than existing methods?
• RQ3 How does the performance of ASR models fine-tuned on test cases selected by Prophet compare to those fine-tuned on baselines?
• RQ4 Which features of a test case suite are most strongly correlated with improvements in ASR model performance?
• RQ5 Is the original model's WER on a test suite a better predictor of fine-tuning effectiveness than phonetic or triphone richness?

Key findings

• Prophet achieves a 12.63% higher word error rate (WER) on test suites it prioritizes compared to the second-best method, indicating significantly more errors uncovered under the same budget.
• Fine-tuning ASR models using test cases selected by Prophet results in significantly larger performance improvements than using test cases selected by four baseline methods.
• The original model’s WER on a test suite is the strongest predictor of fine-tuning effectiveness, outperforming features like phoneme richness or triphone diversity.
• Prophet’s approach, which focuses on error-proneness prediction, outperforms methods that prioritize phonetic diversity, demonstrating that error potential is a better signal than linguistic coverage.
• Statistical analysis confirms that test cases with higher predicted error scores lead to more effective model fine-tuning, validating the method’s design.
• The replication package is publicly released to support reproducibility and future research.