[Paper Review] A Closer Look into Mixture-of-Experts in Large Language Models

Ka Man Lo, Zeyu Huang+3 more|arXiv (Cornell University)|Jun 26, 2024

Computational and Text Analysis Methods4 citations

TL;DR

This paper provides a comprehensive empirical analysis of Mixture-of-Experts (MoE) in large language models, examining parameter and behavioral characteristics across three models—Mixtral 8x7B, DeepSeekMoE, and Grok-1. It reveals that neurons act as fine-grained experts, routers favor experts with higher output norms, and expert diversity increases with depth, with the final layer being an outlier. The findings suggest training from scratch enhances expert diversity and inform router design and expert allocation strategies.

ABSTRACT

Mixture-of-experts (MoE) is gaining increasing attention due to its unique properties and remarkable performance, especially for language tasks. By sparsely activating a subset of parameters for each token, MoE architecture could increase the model size without sacrificing computational efficiency, achieving a better trade-off between performance and training costs. However, the underlying mechanism of MoE still lacks further exploration, and its modularization degree remains questionable. In this paper, we make an initial attempt to understand the inner workings of MoE-based large language models. Concretely, we comprehensively study the parametric and behavioral features of three popular MoE-based models and reveal some intriguing observations, including 1) Neurons act like fine-grained experts; 2) The router of MoE usually selects experts with larger output norms; 3) The expert diversity increases as the layer increases, while the last layer is an outlier, which is further validated by an initial experiment. Based on the observations, we also provide suggestions for a broad spectrum of MoE practitioners, such as router design and expert allocation. We hope this work could shed light on future research on the MoE framework and other modular architectures. Code is available at https://github.com/kamanphoebe/Look-into-MoEs.

Motivation & Objective

• To investigate whether MoE-based LLMs achieve true modularization through heterogeneous expert specialization or remain homogeneous ensembles.
• To analyze the parametric and behavioral features of experts in recent MoE models, focusing on parameter correlation, routing behavior, and output similarity.
• To identify architectural and training-level factors that influence expert diversity and specialization in MoE models.
• To provide actionable insights for router design, expert allocation, and training schemes based on empirical observations.

Proposed method

• Empirical analysis of three recent MoE models: Mixtral 8x7B, DeepSeekMoE, and Grok-1, using their publicly available weights.
• Computation of expert parameter similarity via cosine similarity between weight matrices and output feature vectors across diverse inputs.
• Examination of router behavior by correlating gate scores with expert output norms and activation patterns.
• Visualization and analysis of neuron-level activation heatmaps to assess fine-grained expert behavior and similarity across experts.
• Comparison of models trained via different schemes—specifically, Mixtral (likely via upcycling) vs. DeepSeek and Grok-1 (trained from scratch)—to assess impact on expert correlation.
• Use of norm-based routing supervision and correlation measurement to evaluate routing efficiency and expert distinctiveness.

Figure 5: Intermediate state values of Mixtral experts. The top $k$ experts are shown on top of each heat map. Each number in the vertical axis refers to an expert index while the horizontal axis represents the number of neurons.

Experimental results

Research questions

• RQ1 Do experts in MoE-based LLMs exhibit meaningful heterogeneity in parameters and behavior, or are they highly correlated?
• RQ2 How does the router in MoE models select experts, and is there a preference for experts with higher output norms?
• RQ3 How does expert diversity change across model depth, and why does the final layer show a reversal in similarity trends?
• RQ4 To what extent does training from scratch versus specialized initialization affect expert diversity and correlation?
• RQ5 Can weight matrix similarity serve as a proxy for output feature similarity across experts?

Key findings

• Neurons within the feed-forward network act as fine-grained experts, with gate embeddings and gate projection matrices showing correlated activation patterns, suggesting neuron-level specialization.
• The router in Mixtral 8x7B and DeepSeekMoE consistently selects experts with higher output norms, indicating norm-based routing is a robust and effective strategy.
• Expert similarity decreases with increasing layer depth, indicating growing diversity, but the final layer shows a sharp increase in similarity, making it an outlier in the hierarchy.
• Measuring similarity between expert weight matrices correlates strongly with average output similarity across tokens, suggesting weight-based analysis can efficiently proxy output-level evaluation.
• Models trained from scratch (e.g., DeepSeekMoE, Grok-1) exhibit weaker correlations between expert parameters and behaviors compared to Mixtral, implying training from scratch may better promote expert diversity.
• The strong parameter and behavioral correlations observed in Mixtral suggest it may have been trained using a specialized initialization scheme such as upcycling from a base model, rather than from scratch.