[Paper Review] A Review of Sparse Expert Models in Deep Learning

William Fedus, Jeff Dean+1 more|arXiv (Cornell University)|Sep 4, 2022

COVID-19 diagnosis using AI33 citations

TL;DR

This paper surveys sparse expert models (e.g., Mixture-of-Experts) in deep learning, detailing architectures, routing mechanisms, scaling laws, and cross-domain applications. It highlights system-level considerations and future directions for sparse experts.

ABSTRACT

Sparse expert models are a thirty-year old concept re-emerging as a popular architecture in deep learning. This class of architecture encompasses Mixture-of-Experts, Switch Transformers, Routing Networks, BASE layers, and others, all with the unifying idea that each example is acted on by a subset of the parameters. By doing so, the degree of sparsity decouples the parameter count from the compute per example allowing for extremely large, but efficient models. The resulting models have demonstrated significant improvements across diverse domains such as natural language processing, computer vision, and speech recognition. We review the concept of sparse expert models, provide a basic description of the common algorithms, contextualize the advances in the deep learning era, and conclude by highlighting areas for future work.

Motivation & Objective

• Explain the concept and history of sparse expert models in deep learning.
• Summarize common architectures (e.g., MoE, Switch Transformers) and their routing mechanisms.
• Discuss scaling properties upstream and downstream, plus hardware and system considerations.
• Highlight cross-domain applications (NLP, CV, speech, multimodal) and emerging trends.
• Identify open challenges and future research directions in sparse expert modeling.

Proposed method

• Describe the evolution of sparse expert models from early MoE work to modern Transformer-based approaches.
• Summarize key routing algorithms (top-k, top-1, BASE layers, RL-based routing) and their trade-offs.
• Discuss scaling analyses, including effective parameter counts (EPC) and token-budget considerations.
• Review hardware co-design and distributed training techniques (data/model/expert parallelism, all2all communication, load balancing).
• Synthesize cross-domain applications and domain-specific routing inputs (text, image patches, spectrograms).

Experimental results

Research questions

• RQ1 What are the defining characteristics and variants of sparse expert models in deep learning?
• RQ2 How do routing algorithms and hardware co-design influence performance, efficiency, and scalability?
• RQ3 What are the observed scaling behaviors upstream and downstream for sparse expert models?
• RQ4 How do sparse expert models perform across domains such as NLP, vision, and speech, and what transfer dynamics emerge?
• RQ5 What are the main open challenges and promising directions for future work in sparse expert architectures.

Key findings

• Sparse expert models decouple parameter count from per-example compute, enabling very large yet efficient models.
• Upstream scaling shows gains on pre-training tasks, with mixed downstream transfer results across tasks and domains.
• Few-shot and fine-tuning scenarios can benefit from sparse experts, with notable gains over dense baselines in several settings.
• System-level advancements (distributed training, communication-efficient routing, and memory management) improve practicality and speedups.
• Across NLP, vision, and speech, sparse expert approaches (e.g., ST-MoE, GLaM) demonstrate competitive or superior performance with reduced FLOPs or energy use.
• Calibration of sparse models improves with scale, often matching or approaching dense models at higher compute budgets.