[论文解读] A Review of Sparse Expert Models in Deep Learning

William Fedus, Jeff Dean+1 more|arXiv (Cornell University)|Sep 4, 2022

COVID-19 diagnosis using AI被引用 33

一句话总结

本论文综述深度学习中的稀疏专家模型（如 Mixture-of-Experts）等，详细介绍架构、路由机制、扩展定律以及跨领域应用。它强调系统层面的考量和稀疏专家未来的方向。

ABSTRACT

Sparse expert models are a thirty-year old concept re-emerging as a popular architecture in deep learning. This class of architecture encompasses Mixture-of-Experts, Switch Transformers, Routing Networks, BASE layers, and others, all with the unifying idea that each example is acted on by a subset of the parameters. By doing so, the degree of sparsity decouples the parameter count from the compute per example allowing for extremely large, but efficient models. The resulting models have demonstrated significant improvements across diverse domains such as natural language processing, computer vision, and speech recognition. We review the concept of sparse expert models, provide a basic description of the common algorithms, contextualize the advances in the deep learning era, and conclude by highlighting areas for future work.

研究动机与目标

• Explain the concept and history of sparse expert models in deep learning.
• Summarize common architectures (e.g., MoE, Switch Transformers) and their routing mechanisms.
• Discuss scaling properties upstream and downstream, plus hardware and system considerations.
• Highlight cross-domain applications (NLP, CV, speech, multimodal) and emerging trends.
• Identify open challenges and future research directions in sparse expert modeling.

提出的方法

• Describe the evolution of sparse expert models from early MoE work to modern Transformer-based approaches.
• Summarize key routing algorithms (top-k, top-1, BASE layers, RL-based routing) and their trade-offs.
• Discuss scaling analyses, including effective parameter counts (EPC) and token-budget considerations.
• Review hardware co-design and distributed training techniques (data/model/expert parallelism, all2all communication, load balancing).
• Synthesize cross-domain applications and domain-specific routing inputs (text, image patches, spectrograms).

实验结果

研究问题

• RQ1 What are the defining characteristics and variants of sparse expert models in deep learning?
• RQ2 How do routing algorithms and hardware co-design influence performance, efficiency, and scalability?
• RQ3 What are the observed scaling behaviors upstream and downstream for sparse expert models?
• RQ4 How do sparse expert models perform across domains such as NLP, vision, and speech, and what transfer dynamics emerge?
• RQ5 What are the main open challenges and promising directions for future work in sparse expert architectures.

主要发现

• Sparse expert models decouple parameter count from per-example compute, enabling very large yet efficient models.
• Upstream scaling shows gains on pre-training tasks, with mixed downstream transfer results across tasks and domains.
• Few-shot and fine-tuning scenarios can benefit from sparse experts, with notable gains over dense baselines in several settings.
• System-level advancements (distributed training, communication-efficient routing, and memory management) improve practicality and speedups.
• Across NLP, vision, and speech, sparse expert approaches (e.g., ST-MoE, GLaM) demonstrate competitive or superior performance with reduced FLOPs or energy use.
• Calibration of sparse models improves with scale, often matching or approaching dense models at higher compute budgets.