[Paper Review] Cross-Language Transfer Learning, Continuous Learning, and Domain Adaptation for End-to-End Automatic Speech Recognition

Jocelyn Huang, Oleksii Kuchaiev+6 more|arXiv (Cornell University)|May 8, 2020

Speech Recognition and Synthesis21 references20 citations

TL;DR

This paper demonstrates that fine-tuning a large, pre-trained English QuartzNet ASR model significantly improves performance across diverse ASR tasks, including cross-lingual transfer (German, Spanish, Russian), domain adaptation (financial earnings calls), and accent variation. The method consistently outperforms training from scratch, even with small target datasets, and accelerates convergence by up to 18x in large-data settings, with the pre-trained model acting as a strong regularizer that enhances generalization.

ABSTRACT

In this paper, we demonstrate the efficacy of transfer learning and continuous learning for various automatic speech recognition (ASR) tasks. We start with a pre-trained English ASR model and show that transfer learning can be effectively and easily performed on: (1) different English accents, (2) different languages (German, Spanish and Russian) and (3) application-specific domains. Our experiments demonstrate that in all three cases, transfer learning from a good base model has higher accuracy than a model trained from scratch. It is preferred to fine-tune large models than small pre-trained models, even if the dataset for fine-tuning is small. Moreover, transfer learning significantly speeds up convergence for both very small and very large target datasets.

Motivation & Objective

• To investigate whether transfer learning from a pre-trained English ASR model improves performance on low-resource languages, accents, and domain-specific speech.
• To evaluate the effectiveness of fine-tuning large pre-trained models versus training from scratch on small or large target datasets.
• To examine catastrophic forgetting in continual learning and the necessity of retaining pre-training data during fine-tuning.
• To assess the impact of model size and learning rate scheduling on transfer learning performance.

Proposed method

• Fine-tune a pre-trained QuartzNet encoder (15x5) on target datasets using a 10x smaller initial learning rate than used in pre-training.
• For non-English languages with different alphabets, initialize a new shallow decoder; otherwise, load the pre-trained decoder.
• Use the NeMo toolkit for all experiments, maintaining consistent hyperparameters across tasks.
• Include a portion of the original pre-training data during fine-tuning to prevent catastrophic forgetting.
• Apply the same transfer learning recipe across cross-lingual, cross-accent, and domain-adaptation tasks.
• Train on large-scale financial data (50k+ hours) and compare convergence speed and final accuracy between pre-trained and from-scratch models.

Experimental results

Research questions

• RQ1 Does fine-tuning a pre-trained English ASR model yield better performance than training from scratch on low-resource languages?
• RQ2 How does model size affect transfer learning performance when fine-tuning on small target datasets?
• RQ3 Can transfer learning significantly accelerate convergence in large-data ASR training?
• RQ4 To what extent does catastrophic forgetting occur when fine-tuning without retaining pre-training data?
• RQ5 Does the same transfer learning recipe generalize across different types of ASR adaptation (cross-lingual, cross-accent, domain-specific)?

Key findings

• Fine-tuning the pre-trained 15x5 QuartzNet model achieved a 23.35% WER on German Common Voice, outperforming the from-scratch model (30.42%) and the smaller 5x3 model (28.61%).
• For Russian, fine-tuning reduced WER from 59.50% (from-scratch) to 32.20% when using the 5D model, showing strong regularization benefits on small data.
• In the financial domain, the pre-trained model reached 90% of its final accuracy 18x faster than a model trained from scratch.
• The best performance was consistently achieved by fine-tuning large pre-trained models, even when the target dataset was an order of magnitude larger than the pre-training data.
• Catastrophic forgetting occurred when pre-training data was excluded during fine-tuning, confirming the need to retain it for continual learning.
• The method worked effectively across all tested scenarios: cross-lingual, cross-accent, and domain adaptation, with consistent improvements over training from scratch.