[Paper Review] Combined Acoustic and Pronunciation Modelling for Non-Native Speech Recognition

Ghazi Bouselmi, Dominique Fohr+1 more|ArXiv.org|Nov 6, 2007

Speech Recognition and Synthesis6 references4 citations

TL;DR

This paper proposes a combined approach of pronunciation modelling and acoustic adaptation to improve non-native English speech recognition. By using phonetic confusion matrices to link L1 and L2 pronunciations and applying MLLR/MAP adaptation or model re-estimation to acoustic models, the method achieves relative word error rate reductions of 46% to 71% on the HIWIRE database, with model re-estimation combined with pronunciation modelling yielding the best performance.

ABSTRACT

In this paper, we present several adaptation methods for non-native speech recognition. We have tested pronunciation modelling, MLLR and MAP non-native pronunciation adaptation and HMM models retraining on the HIWIRE foreign accented English speech database. The ``phonetic confusion'' scheme we have developed consists in associating to each spoken phone several sequences of confused phones. In our experiments, we have used different combinations of acoustic models representing the canonical and the foreign pronunciations: spoken and native models, models adapted to the non-native accent with MAP and MLLR. The joint use of pronunciation modelling and acoustic adaptation led to further improvements in recognition accuracy. The best combination of the above mentioned techniques resulted in a relative word error reduction ranging from 46% to 71%.

Motivation & Objective

• To improve automatic speech recognition performance for non-native English speakers, who are often poorly handled by standard ASR systems trained on native speech.
• To address the challenge of foreign accents in real-world applications, such as air traffic control, where non-native speakers use English.
• To develop a scalable method that does not require large, dedicated non-native speech corpora for each L1/L2 pair.
• To investigate the effectiveness of combining pronunciation modelling with acoustic adaptation techniques like MLLR, MAP, and model re-estimation.
• To evaluate whether using acoustically adapted native-language models in pronunciation modelling improves recognition accuracy.

Proposed method

• The method uses a 'phonetic confusion' scheme that maps each spoken language (SL) phone to multiple sequences of confused native language (NL) phones based on pronunciation similarities.
• Acoustic models are adapted to non-native accents using MLLR (Maximum Likelihood Linear Regression) and MAP (Maximum A Posteriori) techniques on non-native speech data.
• The approach combines acoustic adaptation with pronunciation modelling by using both canonical SL models and NL models (either canonical or adapted) in the confusion matrix.
• Model re-estimation is applied to SL models using a small adaptation corpus to better capture non-native accent characteristics.
• The lexicon is modified to include alternative pronunciations derived from the confusion matrix, enabling recognition of non-native variants.
• Experiments compare various combinations of HMM model sets—canonical, adapted, or re-estimated—using both constrained and free-word-loop grammars on the HIWIRE database.

Experimental results

Research questions

• RQ1 Can combining pronunciation modelling with acoustic adaptation significantly reduce word error rates in non-native speech recognition?
• RQ2 Does using acoustically adapted native-language models in pronunciation modelling lead to better performance than using canonical models?
• RQ3 How do different adaptation techniques—MLLR, MAP, and model re-estimation—affect recognition accuracy for non-native accents?
• RQ4 Is the performance improvement dependent on the type of grammar used (constrained vs. free-word-loop)?
• RQ5 What is the impact of model variability in the confusion matrix on recognition accuracy?

Key findings

• The joint use of pronunciation modelling and acoustic adaptation reduced word error rate by 46% to 71% relative to the baseline system.
• The best performance was achieved with the 'Confusion7' configuration, which combined model re-estimation of canonical English models with pronunciation modelling using a confusion matrix between canonical English and re-estimated models.
• Model re-estimation outperformed MLLR and MAP adaptation, especially in the constrained grammar condition, achieving a 1.4% WER and 3.2% SER with free grammar.
• Using identical models (e.g., canonical vs. canonical) in the confusion matrix led to worse performance than using diverse models, indicating that model variability enhances recognition.
• The 'Confusion6' system, which used MAP-adapted English models in both confusion sets, outperformed 'Confusion5', which used MAP-adapted native models, likely due to the supervised nature of the English model adaptation.
• The 'Confusion1' and 'Confusion2' systems performed worse than the baseline with MAP adaptation, indicating that confusion between identical or highly similar models reduces system robustness.