[Paper Review] Joint Modeling of Accents and Acoustics for Multi-Accent Speech Recognition

Xuesong Yang, Kartik Audhkhasi+4 more|arXiv (Cornell University)|Feb 7, 2018

Speech Recognition and Synthesis20 references5 citations

TL;DR

This paper proposes a joint end-to-end learning framework that simultaneously trains an acoustic model and an accent identification (AID) model using bidirectional LSTMs with connectionist temporal classification (CTC) loss. By using AID as an auxiliary task during training and applying a hard-switching mechanism based on predicted accents, the model achieves 5.94% relative WER improvement on British English and 9.47% on American English over a strong multi-task baseline, demonstrating that explicit accent supervision enhances both ASR and AID performance.

ABSTRACT

The performance of automatic speech recognition systems degrades with increasing mismatch between the training and testing scenarios. Differences in speaker accents are a significant source of such mismatch. The traditional approach to deal with multiple accents involves pooling data from several accents during training and building a single model in multi-task fashion, where tasks correspond to individual accents. In this paper, we explore an alternate model where we jointly learn an accent classifier and a multi-task acoustic model. Experiments on the American English Wall Street Journal and British English Cambridge corpora demonstrate that our joint model outperforms the strong multi-task acoustic model baseline. We obtain a 5.94% relative improvement in word error rate on British English, and 9.47% relative improvement on American English. This illustrates that jointly modeling with accent information improves acoustic model performance.

Motivation & Objective

• To address the performance degradation in automatic speech recognition (ASR) caused by accent mismatch between training and testing data.
• To investigate whether explicitly modeling accent information during acoustic model training can improve recognition accuracy across multiple accents.
• To explore the mutual benefits of joint training for both multi-accent ASR and accent identification (AID).
• To develop a hard-switching strategy that selects accent-specific output layers based on predicted accent, improving robustness to accent variation.
• To demonstrate that auxiliary AID training enhances the expressiveness of shared acoustic features in a multi-accent setting.

Proposed method

• A bidirectional LSTM (BLSTM) acoustic model is trained with CTC loss to map speech features to phonetic sequences.
• An auxiliary AID network uses average pooling on BLSTM outputs to generate utterance-level accent embeddings for classification.
• The joint model trains the lower layers with both CTC loss (primary task) and AID loss (auxiliary task), with a weighted sum of losses using a hyperparameter α.
• A hard-switching mechanism selects the accent-specific output layer of the CTC model based on the predicted accent from the AID network during inference.
• The model is trained on the Wall Street Journal (American English) and Cambridge (British English) corpora, with accent labels used as supervision for both tasks.
• Optimal trade-off between ASR and AID is found at α = 0.001, minimizing word error rate (WER) and maximizing AID accuracy.

Experimental results

Research questions

• RQ1 Can joint training of acoustic modeling and accent identification improve multi-accent ASR performance compared to standard multi-task learning?
• RQ2 Does using accent prediction as an auxiliary task enhance the learning of accent-invariant and accent-specific features in the shared network layers?
• RQ3 How does the performance of the joint model compare to baseline models when using a hard-switching strategy based on predicted accents?
• RQ4 To what extent does joint training improve the accuracy of the accent identification system itself?
• RQ5 What is the optimal balance between acoustic modeling and accent identification loss during joint training?

Key findings

• The joint model achieves a 5.94% relative word error rate (WER) improvement on British English and 9.47% on American English compared to a strong multi-task acoustic model baseline.
• Oracle performance shows that the joint model outperforms the accent-specific model (ASpec) by 17.97% relative WER improvement on average, and the multi-task model (MTLP) by 6.81%.
• When using an independent AID model for hard-switching, the joint model achieves 22.52% relative WER improvement over ASpec on American English, outperforming the MTLP baseline (14.41% improvement).
• The optimal AID loss weight α = 0.001 yields a minimum WER of 8.9% and an AID accuracy of 97.77%, with high accuracy (>92%) maintained across a wide range of α values.
• The joint model significantly improves AID performance, achieving 97.77% accuracy at α = 0.005, demonstrating that joint training enhances both ASR and AID tasks.
• The results indicate that explicit accent supervision during training enhances feature learning, with shared layers capturing expressive accent-independent representations that refine accent-specific models.