[Paper Review] Multilingual Speech Recognition using Knowledge Transfer across Learning Processes

Rimita Lahiri, Kenichi Kumatani+2 more|arXiv (Cornell University)|Oct 15, 2021

Speech Recognition and Synthesis25 references4 citations

TL;DR

This paper proposes a multilingual speech recognition framework that enhances performance by combining language ID one-hot vectors with a meta-learning approach (LEAP) to transfer knowledge across learning processes, while leveraging self-supervised learning (SSL). The method minimizes expected gradient path length across language-specific task manifolds, achieving a 3.55% relative WER reduction overall and 3.51% with language ID, outperforming baseline SSL and LEAP alone.

ABSTRACT

Multilingual end-to-end(E2E) models have shown a great potential in the expansion of the language coverage in the realm of automatic speech recognition(ASR). In this paper, we aim to enhance the multilingual ASR performance in two ways, 1)studying the impact of feeding a one-hot vector identifying the language, 2)formulating the task with a meta-learning objective combined with self-supervised learning (SSL). We associate every language with a distinct task manifold and attempt to improve the performance by transferring knowledge across learning processes itself as compared to transferring through final model parameters. We employ this strategy on a dataset comprising of 6 languages for an in-domain ASR task, by minimizing an objective related to expected gradient path length. Experimental results reveal the best pre-training strategy resulting in 3.55% relative reduction in overall WER. A combination of LEAP and SSL yields 3.51% relative reduction in overall WER when using language ID.

Motivation & Objective

• To improve multilingual automatic speech recognition (ASR) performance in low-resource and imbalanced language settings.
• To investigate whether conditioning on language ID as a one-hot vector enhances multilingual ASR accuracy.
• To explore knowledge transfer across learning processes via meta-learning, specifically using the LEAP framework to minimize expected gradient path length.
• To integrate self-supervised learning (SSL) with meta-learning for improved cross-lingual representation learning.
• To evaluate the combined effect of language ID, LEAP, and SSL on word error rate (WER) across multiple languages.

Proposed method

• The model uses a Transformer Transducer architecture with a Transformer encoder and LSTM decoder, trained end-to-end.
• Language ID is injected as a one-hot vector input to guide learning per language, improving representation of low-resource languages.
• The LEAP meta-learning framework is applied to optimize initialization by minimizing the expected path length traversed during training across language-specific tasks.
• Self-supervised learning (SSL) pretraining is performed on large-scale unlabeled multilingual data before fine-tuning with labeled data.
• The training pipeline combines iterative self-training and pretraining, with AdamW for fine-tuning and Adam as the meta-optimizer in LEAP-SSL.
• The objective function combines SSL contrastive loss with a path-length regularization term to promote efficient learning across language tasks.

Experimental results

Research questions

• RQ1 Does injecting language ID as a one-hot vector improve multilingual ASR performance, especially for low-resource languages?
• RQ2 Can knowledge transfer across learning processes via meta-learning (LEAP) reduce the expected gradient path length and improve multilingual ASR accuracy?
• RQ3 How does combining LEAP with self-supervised learning (SSL) affect WER across multiple languages?
• RQ4 Does the performance gain from LEAP-SSL depend on the presence of language ID information?
• RQ5 What is the impact of using the same data for SSL pretraining and fine-tuning on multilingual representation learning?

Key findings

• The combination of LEAP and SSL with language ID input yields a 3.51% relative reduction in overall word error rate (WER).
• The use of language ID as a one-hot vector leads to a 21.06% relative WER reduction for Russian (RU) and 20.69% for French (FR-FR), indicating strong gains for specific languages.
• Without language ID, LEAP-SSL leads to a slight performance degradation in most locales, suggesting that language information is critical for effective meta-learning initialization.
• SSL alone improves WER across all languages, especially when language ID is not used, demonstrating its effectiveness in learning general multilingual representations.
• The LEAP-SSL method with language ID achieves consistent WER improvements across all languages except PT-BR, where gains are minimal.
• The overall best pre-training strategy results in a 3.55% relative WER reduction, confirming the effectiveness of the proposed knowledge transfer mechanism.