[論文レビュー] The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications

Pan Dong, BingTao Li+3 more|arXiv (Cornell University)|Feb 8, 2026

Domain Adaptation and Few-Shot Learning被引用数 0

ひとこと要約

sparse MoEモデルの包括的な調査。基盤となるルーティングとエキスパ/Nets、分散学習/推論パラダイム、垂直ドメインの適用、課題と今後の方向性を網羅。

ABSTRACT

The sparse Mixture of Experts(MoE) architecture has evolved as a powerful approach for scaling deep learning models to more parameters with comparable computation cost. As an important branch of large language model(LLM), MoE model only activate a subset of experts based on a routing network. This sparse conditional computation mechanism significantly improves computational efficiency, paving a promising path for greater scalability and cost-efficiency. It not only enhance downstream applications such as natural language processing, computer vision, and multimodal in various horizontal domains, but also exhibit broad applicability across vertical domains. Despite the growing popularity and application of MoE models across various domains, there lacks a systematic exploration of recent advancements of MoE in many important fields. Existing surveys on MoE suffer from limitations such as lack coverage or none extensively exploration of key areas. This survey seeks to fill these gaps. In this paper, Firstly, we examine the foundational principles of MoE, with an in-depth exploration of its core components-the routing network and expert network. Subsequently, we extend beyond the centralized paradigm to the decentralized paradigm, which unlocks the immense untapped potential of decentralized infrastructure, enables democratization of MoE development for broader communities, and delivers greater scalability and cost-efficiency. Furthermore we focus on exploring its vertical domain applications. Finally, we also identify key challenges and promising future research directions. To the best of our knowledge, this survey is currently the most comprehensive review in the field of MoE. We aim for this article to serve as a valuable resource for both researchers and practitioners, enabling them to navigate and stay up-to-date with the latest advancements.

研究の動機と目的

• Explain the foundational design of sparse MoE models, focusing on routing networks and expert networks.
• Analyze centralized vs. decentralized training/inference paradigms for MoE systems.
• Explore vertical-domain applications of MoE in sectors like medicine, autonomous driving, and finance.
• Identify key challenges (load balancing, capacity, heterogeneity, privacy) and outline future research directions.

提案手法

• Review and synthesize core MoE components: routing network and expert network.
• Discuss routing mechanisms including token choice routing and expert choice routing with load-balancing considerations.
• Describe expert network innovations such as shared experts and interpretability aspects.
• Contrast centralized and decentralized paradigms, including hardware, communication, and fault-tolerance challenges.
• Summarize vertical-domain applications and relevant frameworks that support MoE deployment.

実験結果

リサーチクエスチョン

• RQ1 What are the fundamental components and mechanisms of sparse MoE models (routing vs. expert networks) and how do they interact?
• RQ2 How do centralized and decentralized paradigms affect scalability, resource utilization, and accessibility of MoE systems?
• RQ3 What vertical-domain applications have MoE approaches been explored in, and what are their associated requirements and challenges?
• RQ4 What are the major challenges (e.g., load balancing, expert capacity, heterogeneity, privacy) and future directions for MoE research and deployment?

主な発見

• The survey presents MoE as a scalable approach enabling larger parameter sizes at similar computational budgets.
• It analyzes routing mechanisms (token choice vs. expert choice) and load-balancing strategies, including expert-level and device-level auxiliary losses and expert capacity concepts.
• It introduces decentralized MoE concepts and highlights challenges and research efforts for heterogeneous hardware, limited bandwidth, fault tolerance, and security/privacy.
• It covers centralized frameworks and hybrid parallelism techniques (data, pipeline, tensor/Mei) and discusses expert parallelism and asynchronous approaches.
• It surveys vertical-domain applications such as medical diagnosis, autonomous driving, finance, and business intelligence, emphasizing the breadth of MoE applicability.