[Paper Review] Multilingual Speech Recognition With A Single End-To-End Model

Shubham Toshniwal, Tara N. Sainath+5 more|arXiv (Cornell University)|Nov 6, 2017

Speech Recognition and Synthesis10 references17 citations

TL;DR

This paper proposes a single end-to-end sequence-to-sequence model for multilingual speech recognition across 9 Indian languages with minimal script overlap. By training jointly on a union of grapheme sets without explicit language supervision, the model achieves a 21% relative WER reduction over monolingual baselines and further improves by 7% with language-conditional encoder input, demonstrating strong cross-lingual generalization and implicit language disentanglement.

ABSTRACT

Training a conventional automatic speech recognition (ASR) system to support multiple languages is challenging because the sub-word unit, lexicon and word inventories are typically language specific. In contrast, sequence-to-sequence models are well suited for multilingual ASR because they encapsulate an acoustic, pronunciation and language model jointly in a single network. In this work we present a single sequence-to-sequence ASR model trained on 9 different Indian languages, which have very little overlap in their scripts. Specifically, we take a union of language-specific grapheme sets and train a grapheme-based sequence-to-sequence model jointly on data from all languages. We find that this model, which is not explicitly given any information about language identity, improves recognition performance by 21% relative compared to analogous sequence-to-sequence models trained on each language individually. By modifying the model to accept a language identifier as an additional input feature, we further improve performance by an additional 7% relative and eliminate confusion between different languages.

Motivation & Objective

• To address the challenge of training multilingual ASR systems with limited data and language-specific components.
• To explore whether a unified sequence-to-sequence model can jointly learn multiple Indian languages despite significant script and phonetic differences.
• To evaluate the impact of language identity conditioning on model performance and language confusion.
• To investigate whether the model can perform code-switching between languages and understand the dominance of language modeling over acoustic modeling.

Proposed method

• Training a single attention-based sequence-to-sequence model on a union of language-specific grapheme sets from 9 Indian languages.
• Using a Listen, Attend and Spell (LAS) architecture with bidirectional RNN encoders and unidirectional RNN decoders.
• Introducing a language identifier as an additional input feature to condition the encoder, enabling language-aware modeling.
• Applying attention mechanisms to align acoustic features with predicted grapheme sequences using context vectors derived from encoder states.
• Employing stacked 80-dimensional log-mel features with frame stacking and striding to reduce sequence length.
• Evaluating model behavior via synthetic data, language mismatch experiments, and code-switching tests.

Experimental results

Research questions

• RQ1 How often does the joint multilingual model confuse between languages when no language identity is provided?
• RQ2 Can the model perform code-switching between Indian languages not seen together during training?
• RQ3 How does conditioning the encoder on language identity affect recognition accuracy and language fidelity?
• RQ4 To what extent does the language model dominate the acoustic model in the joint model?
• RQ5 Does the model learn a disentangled representation of acoustic-phonetic content from language identity?

Key findings

• The joint multilingual model reduces WER by 21% relative compared to monolingual models trained independently on each language.
• Adding language identity conditioning improves performance by an additional 7% relative, eliminating language confusion.
• The model rarely produces mixed-script outputs, indicating strong implicit language identification even without explicit supervision.
• The model fails to perform code-switching between Tamil and Hindi in synthetic mixed-language utterances, suggesting language modeling dominates acoustic modeling.
• When given a Hindi language ID for Urdu speech, the model transliterates the audio into Hindi script, indicating disentanglement of phonetic content from language identity.
• The encoder-conditioned model is highly faithful to the provided language ID, even when the input speech is from a mismatched language.