[Paper Review] SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts

Zhao You, Shulin Feng+2 more|arXiv (Cornell University)|May 7, 2021

Speech Recognition and Synthesis26 references5 citations

TL;DR

This paper proposes SpeechMoE, a dynamic routing Mixture of Experts (MoE) architecture for large-scale acoustic models in speech recognition. By integrating a shared embedding network with router inputs and introducing a sparsity L1 loss and mean importance loss, SpeechMoE achieves 7.0% to 23.0% relative character error rate (CER) improvements over strong baselines while maintaining comparable FLOPs, demonstrating superior capacity and efficiency through expert specialization and balanced routing.

ABSTRACT

Recently, Mixture of Experts (MoE) based Transformer has shown promising results in many domains. This is largely due to the following advantages of this architecture: firstly, MoE based Transformer can increase model capacity without computational cost increasing both at training and inference time. Besides, MoE based Transformer is a dynamic network which can adapt to the varying complexity of input instances in realworld applications. In this work, we explore the MoE based model for speech recognition, named SpeechMoE. To further control the sparsity of router activation and improve the diversity of gate values, we propose a sparsity L1 loss and a mean importance loss respectively. In addition, a new router architecture is used in SpeechMoE which can simultaneously utilize the information from a shared embedding network and the hierarchical representation of different MoE layers. Experimental results show that SpeechMoE can achieve lower character error rate (CER) with comparable computation cost than traditional static networks, providing 7.0%-23.0% relative CER improvements on four evaluation datasets.

Motivation & Objective

• To address the challenge of scaling acoustic models without increasing computational cost in speech recognition.
• To improve model capacity and adaptivity by enabling dynamic routing of inputs to specialized experts based on input complexity.
• To enhance routing sparsity and expert utilization balance through novel training objectives.
• To evaluate the effectiveness of combining shared embeddings with layer outputs for improved router decision-making in low-level speech features.

Proposed method

• Proposes a new MoE architecture, SpeechMoE, where the router uses both the previous layer’s output and a shared embedding network as input to improve routing decisions.
• Introduces a sparsity L1 loss to encourage sparse router activation per input, promoting efficient expert utilization.
• Employs a mean importance loss to balance expert usage and prevent underutilization, enhancing model diversity and robustness.
• Uses a top-1 routing strategy similar to Switch Transformer, activating only one expert per input token based on router scores.
• Employs a shared embedding network to transform low-level spectrogram features into higher-level representations for better routing guidance.
• Trains the model using a composite loss combining CTC loss, sparsity L1 loss, and mean importance loss to jointly optimize accuracy, sparsity, and balance.

Figure 1: (a), (b) and (c) represent the architecture of DeepMoE, Switch Transformer and SpeechMoE respectively. Similar to Switch Transformer, only one expert with the largest router probability in each MoE layer is used in the SpeechMoE, which is different from DeepMoE. Besides, the SpeechMoE util

Experimental results

Research questions

• RQ1 Can a dynamic routing MoE architecture improve speech recognition performance while maintaining fixed FLOPs?
• RQ2 How does combining shared embeddings with layer outputs affect router performance in low-level speech features?
• RQ3 To what extent does the proposed sparsity L1 loss improve routing sparsity and model efficiency?
• RQ4 How does the mean importance loss enhance expert utilization balance and model generalization?
• RQ5 What is the impact of increasing the number of experts on model performance and training dynamics?

Key findings

• SpeechMoE with sparsity L1 and mean importance losses achieves 23.0% relative CER improvement on the Read test set compared to the baseline model.
• The model with 8 experts (MoE-8e) achieves 11.9% relative CER reduction on the AISHELL test set, demonstrating strong gains with increased capacity.
• Adding a shared embedding network to the router input reduces CER by 0.19 points on the Spon set and 0.49 points on the Chat set, showing its effectiveness in low-level feature routing.
• The mean importance loss further reduces CER by 0.15 points on the Spon set compared to the baseline MoE model, indicating improved expert balance.
• Increasing the number of experts from 2 to 8 leads to lower validation CTC loss and faster training, confirming scalability and performance gains.
• The model maintains comparable FLOPs (2.3B) across all configurations, proving that increased capacity does not incur additional computational cost.

Figure 2: Validation CTC loss for increasing expert number