[Paper Review] Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model

Anjuli Kannan, Arindrima Datta+7 more|arXiv (Cornell University)|Sep 11, 2019

Speech Recognition and Synthesis32 references4 citations

TL;DR

This paper presents a streaming end-to-end (E2E) multilingual speech recognition model using RNN-T that outperforms both monolingual E2E and conventional ASR systems across nine Indian languages. By combining language vectors and language-specific adapter modules, the model achieves a consistent 10% relative WER reduction—up to 34% on low-resource languages—while maintaining low latency for real-time applications.

ABSTRACT

Multilingual end-to-end (E2E) models have shown great promise in expansion of automatic speech recognition (ASR) coverage of the world's languages. They have shown improvement over monolingual systems, and have simplified training and serving by eliminating language-specific acoustic, pronunciation, and language models. This work presents an E2E multilingual system which is equipped to operate in low-latency interactive applications, as well as handle a key challenge of real world data: the imbalance in training data across languages. Using nine Indic languages, we compare a variety of techniques, and find that a combination of conditioning on a language vector and training language-specific adapter layers produces the best model. The resulting E2E multilingual model achieves a lower word error rate (WER) than both monolingual E2E models (eight of nine languages) and monolingual conventional systems (all nine languages).

Motivation & Objective

• To develop a streaming, low-latency multilingual ASR system suitable for interactive applications like speech assistants.
• To address data imbalance in multilingual training, where high-resource languages dominate due to skewed data distributions.
• To improve performance on low-resource languages without sacrificing accuracy on high-resource ones.
• To demonstrate that a single E2E model can outperform multiple monolingual conventional systems while simplifying deployment.

Proposed method

• Uses the RNN-T architecture for streaming inference, enabling real-time latency by conditioning predictions on past outputs and current acoustic frames.
• Employs a language vector to condition the model on the target language, improving disambiguation across languages.
• Applies language-specific adapter modules—small, trainable feedforward networks inserted into encoder layers—to allow parameter-efficient specialization per language.
• Trains with class-balanced sampling and adaptive upweighting to mitigate data imbalance, though this is less effective when combined with language vectors.
• Combines language vectors and adapters to jointly improve robustness and performance across all languages, especially low-resource ones.
• Evaluates the model on a 37,000-hour multilingual corpus of nine Indian languages, comparing against monolingual E2E and conventional ASR baselines.

Experimental results

Research questions

• RQ1 Can a streaming end-to-end multilingual ASR model achieve competitive performance with state-of-the-art monolingual conventional systems while maintaining low latency?
• RQ2 How effective are language vectors and adapter modules in mitigating performance degradation on low-resource languages in imbalanced multilingual training?
• RQ3 Does combining language vectors with language-specific adapter modules yield better results than either technique alone or data sampling strategies?
• RQ4 To what extent does data imbalance affect model performance, and can architectural modifications like adapters compensate for skewed data distributions?
• RQ5 Can a single E2E model replace nine separate monolingual models in terms of accuracy and deployment efficiency?

Key findings

• The proposed multilingual E2E model achieves a lower word error rate (WER) than monolingual E2E models on eight of nine languages.
• The model outperforms monolingual conventional systems on all nine languages, with a consistent 10% relative WER reduction, and up to 34% relative improvement on low-resource languages like Kannada.
• The combination of language vectors and adapter modules yields the best performance, with 9% relative WER reduction on Kannada and 8% on Urdu.
• Language vector conditioning alone reduces WER on low-resource languages but causes regression on high-resource ones; adapters mitigate this by enabling language-specific adaptation without overfitting.
• Adapters add only 2.5 million parameters per language (about 2% of the full model), making them highly parameter-efficient and deployable selectively.
• The model maintains streaming capability with RNN-T, enabling real-time inference suitable for interactive applications like speech assistants.