[論文レビュー] You don't understand me!: Comparing ASR results for L1 and L2 speakers of Swedish

Ronald Cumbal, Birger Moëll+2 more|arXiv (Cornell University)|May 22, 2024

Speech and dialogue systems被引用数 7

ひとこと要約

この論文は、3つのASRサービス（Google、Microsoft、Huggingface）を用いて、読み上げと自然発話におけるL1とL2のスウェーデン語話者間のASR性能を比較し、誤りタイプと発話長の影響を分析する。

ABSTRACT

The performance of Automatic Speech Recognition (ASR) systems has constantly increased in state-of-the-art development. However, performance tends to decrease considerably in more challenging conditions (e.g., background noise, multiple speaker social conversations) and with more atypical speakers (e.g., children, non-native speakers or people with speech disorders), which signifies that general improvements do not necessarily transfer to applications that rely on ASR, e.g., educational software for younger students or language learners. In this study, we focus on the gap in performance between recognition results for native and non-native, read and spontaneous, Swedish utterances transcribed by different ASR services. We compare the recognition results using Word Error Rate and analyze the linguistic factors that may generate the observed transcription errors.

研究の動機と目的

• Assess the Word Error Rate (WER) gap between native (L1) and non-native (L2) Swedish speech for read and spontaneous speech.
• Evaluate multiple off-the-shelf ASR systems on Swedish under non-ideal conditions.
• Identify common transcription errors and linguistic factors contributing to misrecognitions.
• Examine how utterance length affects ASR performance for L1 vs L2 Swedish speech.
• Discuss implications for educational and language-learning applications using ASR.

提案手法

• Use two Swedish L2 datasets (Ville read sentences; CORALL social conversations) with native and non-native speakers.
• Test three ASR systems: Google Cloud Speech-to-Text, Microsoft Azure Speech-to-Text, and Huggingface wav2vec2-based model.
• Measure performance with Word Error Rate (WER) and Number of Samples Failed to Recognize (NFR).
• Segment results by utterance length (short, medium, long) to analyze length effects.
• Analyze transcription errors to identify frequently misrecognized words and categories (deletions vs substitutions).
• Perform statistical tests (Welch’s t-test) to assess significance of native vs non-native differences.

実験結果

リサーチクエスチョン

• RQ1 Does the native vs non-native speech gap in ASR performance persist across read and spontaneous Swedish speech?
• RQ2 How do different ASR services compare in handling L1 vs L2 Swedish?
• RQ3 What are the common error patterns for non-native Swedish speech, and do they differ from native speech?
• RQ4 How does utterance length influence ASR performance for L1 and L2 speech?
• RQ5 What are the implications of ASR weaknesses for educational or language-learning applications?

主な発見

データセット	話者タイプ	Google WER	Microsoft WER	Huggingface WER
Ville（読み上げ文）	ネイティブ	0.162	0.111	0.522
Ville（読み上げ文）	非ネイティブ	0.325	0.410	0.593
CORALL（ソーシャル会話）	ネイティブ	0.412	0.356	0.641
CORALL（ソーシャル会話）	非ネイティブ	0.421	0.507	0.663

• Native speakers generally achieve lower WER than non-native speakers, with the gap more pronounced in read sentences and less so in spontaneous speech for some ASRs.
• Microsoft Azure showed a significant native vs non-native difference in spontaneous speech (N: 0.36 vs NN: 0.51, p<0.05).
• Google Cloud and Huggingface did not show a statistically significant native vs non-native difference in spontaneous speech for the datasets studied.
• In read sentences, longer utterances generally had better WER for native speakers, but effects were mixed for non-native speakers and varied by ASR.
• Spontaneous speech often resulted in many non-recognized short utterances (NFR), particularly for Google and Microsoft, affecting usability in interactive educational contexts.
• Common misrecognitions include short function words (e.g., ja, och, du, jag) and learner-specific terms (e.g., förstår, repetera), highlighting language-learning signaling words as error-prone.