[Paper Review] Transfer learning of language-independent end-to-end ASR with language model fusion

Hirofumi Inaguma, Jaejin Cho+3 more|arXiv (Cornell University)|Nov 6, 2018

Speech Recognition and Synthesis19 references6 citations

TL;DR

This paper proposes LM fusion transfer, a method that integrates an external language model into the decoder of a pre-trained, language-independent end-to-end ASR system during adaptation to low-resource languages. By leveraging external text data via cold fusion during fine-tuning, the approach significantly improves ASR performance—achieving relative gains of up to 21.6% over prior transfer learning methods and closing the performance gap with state-of-the-art hybrid systems on low-resource data (e.g., ~50 hours).

ABSTRACT

This work explores better adaptation methods to low-resource languages using an external language model (LM) under the framework of transfer learning. We first build a language-independent ASR system in a unified sequence-to-sequence (S2S) architecture with a shared vocabulary among all languages. During adaptation, we perform LM fusion transfer, where an external LM is integrated into the decoder network of the attention-based S2S model in the whole adaptation stage, to effectively incorporate linguistic context of the target language. We also investigate various seed models for transfer learning. Experimental evaluations using the IARPA BABEL data set show that LM fusion transfer improves performances on all target five languages compared with simple transfer learning when the external text data is available. Our final system drastically reduces the performance gap from the hybrid systems.

Motivation & Objective

• Address the challenge of low-resource automatic speech recognition (ASR) by improving adaptation to new languages with limited transcribed data.
• Investigate whether linguistic context from external language models (LMs) enhances transfer learning in end-to-end ASR systems.
• Explore the effectiveness of different LM fusion strategies—shallow, deep, and cold fusion—during the adaptation phase of a multilingual S2S model.
• Evaluate the impact of seed model capacity and multilingual training data diversity on low-resource adaptation performance.
• Demonstrate that integrating external text data via cold fusion transfer can rival or surpass conventional hybrid ASR systems in low-resource settings.

Proposed method

• Train a language-independent end-to-end ASR system using a unified sequence-to-sequence (S2S) architecture with a shared vocabulary across multiple languages.
• Perform transfer learning by fine-tuning the pre-trained S2S model on target low-resource languages using only speech data (initial transfer step).
• Introduce LM fusion transfer: integrate an external RNNLM into the decoder network during the adaptation phase using cold fusion, where the LM is fused throughout the entire adaptation training stage.
• Use external text data (e.g., from IARPA BABEL) to pre-train the RNNLM, which is then fused with the S2S decoder to improve language modeling during inference and training.
• Compare multiple fusion strategies—shallow fusion, deep fusion, and cold fusion—by evaluating their performance on five low-resource languages with varying amounts of speech data.
• Assess the impact of model capacity and multilingual training data size (600–2200 hours) on adaptation performance, particularly when combined with external text data.

Fig. 1 : Overview of language model fusion transfer. LM fusion transfer is conducted with monolingual data only.

Experimental results

Research questions

• RQ1 Can linguistic context from an external language model improve transfer learning performance in low-resource end-to-end ASR?
• RQ2 How does LM fusion transfer compare to simple transfer learning with shallow fusion in terms of WER reduction on low-resource languages?
• RQ3 Does the effectiveness of LM fusion depend on the availability and size of external text data?
• RQ4 How does the choice of seed model (trained on varying amounts and varieties of multilingual data) affect adaptation performance?
• RQ5 Can cold fusion transfer outperform other fusion methods (shallow, deep) when external text data is available during adaptation?

Key findings

• LM fusion transfer significantly improves ASR performance on all five low-resource languages (Assamese, Swahili, Lao, Tagalog, Zulu) compared to simple transfer learning with shallow fusion when external text data is available.
• Cold fusion transfer (CF-transfer) achieved a relative 21.6% WER reduction on average compared to prior transfer learning methods on the FLP (~50h) data set, with gains of up to 6.8% over transfer learning without LM fusion.
• On the LLP (~10h) data set, CF-transfer outperformed shallow fusion by 10.4% relatively on average when using larger external text data (FLP), demonstrating the value of linguistic context in adaptation.
• The performance gap between the proposed end-to-end system and state-of-the-art BLSTM-HMM hybrid systems was drastically reduced, especially when external text data was used.
• Even with only 10 hours of speech data, WER dropped significantly when using an RNNLM trained on 50 hours of text, indicating that linguistic context is highly beneficial in low-resource settings.
• The method is efficient and practical: integrating the external LM only during adaptation incurs minimal computational cost, making it suitable for rapid deployment in new languages.