[Paper Review] Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metric

Suyoun Kim, Manh Duc Le+7 more|arXiv (Cornell University)|Oct 11, 2021

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes SemDist, a semantic distance metric based on pre-trained language model embeddings to evaluate automatic speech recognition (ASR) output quality by measuring semantic correctness rather than word-level errors. It demonstrates that SemDist correlates significantly higher with human user perception and downstream NLU tasks than Word Error Rate (WER), with pairwise-token-based XLM embeddings yielding the strongest performance on 71K and 36K user-annotated ASR evaluations.

ABSTRACT

Measuring automatic speech recognition (ASR) system quality is critical for creating user-satisfying voice-driven applications. Word Error Rate (WER) has been traditionally used to evaluate ASR system quality; however, it sometimes correlates poorly with user perception/judgement of transcription quality. This is because WER weighs every word equally and does not consider semantic correctness which has a higher impact on user perception. In this work, we propose evaluating ASR output hypotheses quality with SemDist that can measure semantic correctness by using the distance between the semantic vectors of the reference and hypothesis extracted from a pre-trained language model. Our experimental results of 71K and 36K user annotated ASR output quality show that SemDist achieves higher correlation with user perception than WER. We also show that SemDist has higher correlation with downstream Natural Language Understanding (NLU) tasks than WER.

Motivation & Objective

• To address the poor correlation between traditional WER and human user perception of ASR quality.
• To evaluate whether semantic correctness, measured via pre-trained language model embeddings, better reflects user judgment than word-level error rates.
• To compare multiple strategies for computing semantic distance and identify the most robust method for user perception modeling.
• To demonstrate SemDist's utility in predicting user satisfaction and improving model selection in voice-driven applications.

Proposed method

• Compute semantic embeddings for reference and hypothesis transcriptions using pre-trained transformer-based language models (e.g., RoBERTa, XLM).
• Calculate semantic distance (SemDist) as the cosine distance between the mean-pooled or [CLS]-token embeddings of reference and hypothesis.
• Explore four SemDist variants: RoBERTa-mean-pooling, XLM-mean-pooling, XLM-[CLS], and XLM-pairwise-token embedding methods.
• Use Pearson correlation to compare SemDist and WER against user judgments (UserChoice, UserRating) and downstream NLU tasks (intent accuracy, EM, EMTree).
• Train linear regression models to predict user ratings using WER, SemDist, or both as input features.
• Analyze ranking discrepancies between WER and SemDist to understand their divergent evaluation behaviors.

Experimental results

Research questions

• RQ1 Does SemDist correlate more strongly with human user perception of ASR quality than Word Error Rate (WER)?
• RQ2 Which method of computing semantic embeddings (e.g., mean-pooling, [CLS], pairwise-token) yields the most robust SemDist for user perception?
• RQ3 How does SemDist compare to WER in predicting performance on downstream NLU tasks such as intent recognition and semantic parsing?
• RQ4 Can SemDist be used to train models that accurately predict user satisfaction without requiring costly human annotations?

Key findings

• SemDist achieved significantly higher Pearson correlation with user perception (UserChoice and UserRating) than WER across all tested datasets.
• The XLM-pairwise-token SemDist method showed the highest correlation with user judgment, outperforming RoBERTa-based and mean-pooling variants.
• SemDist correlated more strongly with downstream NLU tasks (1 - IntentAcc, 1 - EM, 1 - EMTree) than WER, indicating better alignment with system-level performance.
• A linear regression model using only SemDist achieved an R² of 0.35 in predicting user ratings, outperforming the WER-only model (R² = 0.12).
• The SemDist-based model achieved a mean absolute error (MAE) of 0.29, matching the best-performing model using both WER and SemDist, indicating SemDist's strong predictive power.
• SemDist effectively penalizes semantically nonsensical errors (e.g., 'cancel an alarm' vs. 'set a alarm') that WER fails to distinguish, highlighting its semantic sensitivity.