[Paper Review] MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models

Leyang Shen, Gongwei Chen+3 more|arXiv (Cornell University)|Jul 17, 2024

Topic Modeling4 citations

TL;DR

This paper proposes MoME, a Mixture of Multimodal Experts that enhances generalist multimodal large language models by specializing vision and language experts to reduce task interference. MoME uses a Mixture of Vision Experts (MoVE) with adaptive deformable transformation for cross-encoder feature alignment and a Mixture of Language Experts (MoLE) with sparsely gated adapters for efficient, low-cost performance gains, achieving up to 20.1-point accuracy gains on document-oriented tasks and strong expert specialization across vision and language modalities.

ABSTRACT

Multimodal large language models (MLLMs) have demonstrated impressive capabilities across various vision-language tasks. However, a generalist MLLM typically underperforms compared with a specialist MLLM on most VL tasks, which can be attributed to task interference. In this paper, we propose a mixture of multimodal experts (MoME) to mitigate task interference and obtain a generalist MLLM. Our MoME is composed of two key components, a mixture of vision experts (MoVE) and a mixture of language experts (MoLE). MoVE can adaptively modulate the features transformed from various vision encoders, and has a strong compatibility in transformation architecture. MoLE incorporates sparsely gated experts into LLMs to achieve painless improvements with roughly unchanged inference costs. In response to task interference, our MoME specializes in both vision and language modality to adapt to task discrepancies. Extensive experiments show that MoME significantly improves the performance of generalist MLLMs across various VL tasks. The source code is released at https://github.com/JiuTian-VL/MoME

Motivation & Objective

• To address task interference in generalist multimodal large language models (MLLMs), which hinders performance across diverse vision-language (VL) tasks.
• To improve generalist MLLM performance by enabling specialization in both vision and language modalities, rather than focusing only on textual differences.
• To design a lightweight, efficient architecture that integrates multimodal experts without significantly increasing inference costs.
• To demonstrate that expert specialization emerges naturally in both vision and language spaces through routing mechanisms.

Proposed method

• Introduces a Mixture of Vision Experts (MoVE) that uses an adaptive deformable transformation (ADT) module to unify and enhance features from multiple vision encoders (e.g., CLIP-ViT, DINOv2, Pix2Struct).
• Employs an instance-level soft router in MoVE to dynamically select and aggregate vision features based on input instruction, enabling task-specific visual representation learning.
• Designs a Mixture of Language Experts (MoLE) that integrates sparsely gated, parameter-efficient adapters into each feed-forward layer of the LLM, minimizing computational overhead.
• Uses a router mechanism in MoLE to route tokens to the most relevant experts per input, enabling dynamic, task-aware language understanding.
• Applies a unified training objective on a diverse instruction-tuning dataset spanning four VL task groups to encourage expert specialization.
• Employs load balancing and routing visualization to validate that experts specialize in distinct task groups, such as Pix2Struct for document tasks.

Experimental results

Research questions

• RQ1 Can a generalist MLLM achieve performance comparable to specialist models by mitigating task interference through multimodal expert specialization?
• RQ2 Does joint specialization in both vision and language modalities lead to better performance across diverse VL tasks than specialization in language alone?
• RQ3 Can a Mixture of Multimodal Experts (MoME) maintain low inference cost while significantly improving multimodal understanding across multiple task groups?
• RQ4 Do the vision and language experts in MoME exhibit clear, task-specific routing patterns that indicate effective specialization?

Key findings

• MoVE achieves an average performance gain of 12.87 points across all VL tasks, with over 20.1-point improvement on the 'Document' group.
• MoLE enables consistent performance gains with minimal computational cost, demonstrating that parameter-efficient adapters can be effectively used in MoE-based MLLMs.
• Visualization of routing results shows strong expert specialization: for example, the 'Pix2Struct' expert is selected over 70% of the time for document-related tasks.
• The MoVE and MoLE components jointly improve performance across all four task groups, with the highest gains observed in document and visual reasoning tasks.
• MoME outperforms state-of-the-art generalist and MoE-based MLLMs on key benchmarks such as TextCaps (+130.8), Flickr30K (94.6), and RefCOCO (83.2), demonstrating superior generalization.
• The routing behavior of both MoVE and MoLE is highly task-dependent, with similar tasks showing similar expert selection patterns, confirming effective specialization.