[Paper Review] Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition

Kenichi Kumatani, Robert Gmyr+6 more|arXiv (Cornell University)|Dec 10, 2021

Speech and Audio Processing4 citations

TL;DR

This paper proposes using sparsely-gated mixture of experts (MoE) to scale up multi-lingual end-to-end speech recognition models with minimal computational overhead. By applying MoE to both sequence-to-sequence Transformer (S2S-T) and Transformer Transducer (T-T) architectures, the method reduces relative word error rates by 16.3% on S2S-T and 4.6% on T-T, demonstrating significant accuracy gains while maintaining efficiency through expert routing.

ABSTRACT

The sparsely-gated Mixture of Experts (MoE) can magnify a network capacity with a little computational complexity. In this work, we investigate how multi-lingual Automatic Speech Recognition (ASR) networks can be scaled up with a simple routing algorithm in order to achieve better accuracy. More specifically, we apply the sparsely-gated MoE technique to two types of networks: Sequence-to-Sequence Transformer (S2S-T) and Transformer Transducer (T-T). We demonstrate through a set of ASR experiments on multiple language data that the MoE networks can reduce the relative word error rates by 16.3% and 4.6% with the S2S-T and T-T, respectively. Moreover, we thoroughly investigate the effect of the MoE on the T-T architecture in various conditions: streaming mode, non-streaming mode, the use of language ID and the label decoder with the MoE.

Motivation & Objective

• To scale multi-lingual end-to-end ASR models for improved accuracy on diverse languages.
• To investigate the effectiveness of sparsely-gated MoE in increasing model capacity without proportional computational cost.
• To compare MoE performance across different architectures: S2S-T and T-T, in both streaming and non-streaming modes.
• To evaluate the impact of MoE on audio encoders versus label decoders, and the role of language ID in routing.
• To explore the scalability of MoE in deeper networks and its potential for knowledge distillation in future work.

Proposed method

• The MoE layer uses a routing network to select only one expert per input token, minimizing computational overhead.
• Each MoE block replaces the feed-forward network (FFN) in Transformer layers with a set of 24–120 sparse experts and a gating mechanism.
• The capacity factor is set to 1.5 to balance expert load and prevent data overflow, with residual connections handling overflows.
• The model uses a simple switching router, where only the top-1 expert is activated per token, reducing computation to near-baseline levels.
• Language ID is incorporated as a one-hot vector to guide routing and improve multi-lingual generalization.
• Experiments are conducted on multiple languages using both streaming and non-streaming inference modes to evaluate robustness.

Experimental results

Research questions

• RQ1 How does MoE scaling improve multi-lingual ASR accuracy on S2S-T and T-T architectures?
• RQ2 What is the relative impact of MoE on different components of the T-T model, such as the audio encoder versus the label decoder?
• RQ3 How does the inclusion of language ID information affect MoE routing and recognition performance?
• RQ4 Does MoE provide greater gains on deeper networks, and how does it compare between streaming and non-streaming inference?
• RQ5 Can MoE improve performance on low-resource languages while maintaining high performance on high-resource ones?

Key findings

• The MoE-S2S-T model achieves a 16.3% relative word error rate reduction compared to the baseline, with 72 experts yielding the best performance.
• The MoE-T-T model reduces WER by 4.6% relatively in non-streaming mode and 4.3% in streaming mode, with gains increasing in deeper architectures.
• Adding MoE to the audio encoder improves recognition accuracy, while applying it to the label decoder yields no significant benefit.
• The use of language ID as input improves WER by up to 14.7% relative in non-MoE T-T models and further enhances MoE performance.
• The deeper T-T model (36 layers) with MoE achieves a 12.18% WER, a 3.2% relative improvement over the non-MoE version, showing scalability.
• The MoE mechanism maintains low computational overhead, with only a marginal increase due to gating, making it efficient for large-scale multi-lingual ASR.