[Paper Review] Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling

Jaejin Cho, Murali Karthick Baskar+7 more|arXiv (Cornell University)|Oct 4, 2018

Speech Recognition and Synthesis18 references5 citations

TL;DR

This paper proposes a multilingual sequence-to-sequence (seq2seq) ASR system using joint CTC-attention training with transfer learning and multilingual RNN language modeling. It demonstrates that fine-tuning a prior multilingual model on low-resource target languages yields significant WER improvements, with RNNLM integration providing gains equivalent to doubling the training data, especially in low-resource settings.

ABSTRACT

Sequence-to-sequence (seq2seq) approach for low-resource ASR is a relatively new direction in speech research. The approach benefits by performing model training without using lexicon and alignments. However, this poses a new problem of requiring more data compared to conventional DNN-HMM systems. In this work, we attempt to use data from 10 BABEL languages to build a multi-lingual seq2seq model as a prior model, and then port them towards 4 other BABEL languages using transfer learning approach. We also explore different architectures for improving the prior multilingual seq2seq model. The paper also discusses the effect of integrating a recurrent neural network language model (RNNLM) with a seq2seq model during decoding. Experimental results show that the transfer learning approach from the multilingual model shows substantial gains over monolingual models across all 4 BABEL languages. Incorporating an RNNLM also brings significant improvements in terms of %WER, and achieves recognition performance comparable to the models trained with twice more training data.

Motivation & Objective

• To investigate transfer learning from a multilingual seq2seq model to low-resource target languages in a joint CTC-attention framework.
• To evaluate the effectiveness of multilingual RNNLM integration during decoding for improving recognition accuracy.
• To analyze the impact of model architecture and data size on transfer learning performance across diverse low-resource languages.
• To compare multilingual transfer learning with monolingual training, focusing on WER and CER improvements.

Proposed method

• A hybrid CTC-attention seq2seq model is trained on 10 BABEL languages using a multi-objective loss combining CTC and attention with a linear interpolation weight λ.
• The encoder uses bidirectional LSTM or deep CNN followed by BLSTM to extract acoustic features, with shared encoder weights for both CTC and attention objectives.
• Transfer learning is performed via stage-2 retraining: the multilingual model is fine-tuned on target languages using varying amounts of data (5–20 hours).
• A character-level RNNLM is trained separately on each target language’s text and used in shallow fusion during decoding via log-score combination: log p = log p_hyp + β log p_lm.
• The joint decoding uses a beam search that combines hypotheses from the seq2seq model and the RNNLM, with β tuned to balance acoustic and language model scores.
• Language modeling is applied post-hoc during inference, not during training, to maintain model flexibility and avoid performance degradation.

Fig. 1 : Hybrid attention/CTC network with LM extension: the shared encoder is trained by both CTC and attention model objectives simultaneously. The joint decoder predicts an output label sequence by the CTC, attention decoder and RNN-LM.

Experimental results

Research questions

• RQ1 Can a multilingual seq2seq model pre-trained on 10 BABEL languages be effectively fine-tuned for 4 unseen low-resource target languages using transfer learning?
• RQ2 How does integrating a multilingual RNNLM during decoding affect recognition performance across varying data sizes?
• RQ3 What is the relative contribution of transfer learning versus language modeling in reducing WER for low-resource ASR?
• RQ4 Does the performance gain from RNNLM integration scale with available training data, or is it most effective in low-data regimes?

Key findings

• Transfer learning from a multilingual prior model reduced WER by an average of 6% across all 4 target languages compared to monolingual models, with consistent gains across data sizes.
• Adding a multilingual RNNLM during decoding reduced WER by approximately 6% on average, with the largest gains observed in low-data regimes (e.g., 5-hour data).
• For Assamese, the WER improvement from RNNLM decoding with 5 hours of data matched the performance of a model retrained on 20 hours of data, demonstrating data efficiency.
• The combination of stage-2 retraining and RNNLM integration achieved a WER of 65.3% on Assamese, 64.3% on Tagalog, 56.2% on Swahili, and 57.9% on Lao—comparable to models trained with twice the data.
• The RNNLM provided consistent WER improvements regardless of the amount of target data used in transfer learning, indicating robustness to data scarcity.
• The joint CTC-attention model with multilingual pretraining and RNNLM decoding outperformed both monolingual models and standalone seq2seq systems, showing strong generalization across diverse low-resource languages.

Fig. 2 : Difference in performance for 5 hours, 10 hours, 20 hours and full set of target language data used to retrain a multilingual model from stage-1