[論文レビュー] VL-Mamba: Exploring State Space Models for Multimodal Learning

Yanyuan Qiao, Zheng Yu+6 more|arXiv (Cornell University)|Mar 20, 2024

Speech and dialogue systems被引用数 10

ひとこと要約

VL-Mamba は、状態空間モデル（Mamba）を backbone とした多模态大規模言語モデルを導入し、Vision Selective Scan に基づく多模态コネクタを搭載、8つのベンチマークで競争力のある結果を達成。

ABSTRACT

Multimodal large language models (MLLMs) have attracted widespread interest and have rich applications. However, the inherent attention mechanism in its Transformer structure requires quadratic complexity and results in expensive computational overhead. Therefore, in this work, we propose VL-Mamba, a multimodal large language model based on state space models, which have been shown to have great potential for long-sequence modeling with fast inference and linear scaling in sequence length. Specifically, we first replace the transformer-based backbone language model such as LLama or Vicuna with the pre-trained Mamba language model. Then, we empirically explore how to effectively apply the 2D vision selective scan mechanism for multimodal learning and the combinations of different vision encoders and variants of pretrained Mamba language models. The extensive experiments on diverse multimodal benchmarks with competitive performance show the effectiveness of our proposed VL-Mamba and demonstrate the great potential of applying state space models for multimodal learning tasks.

研究の動機と目的

• Motivate the use of state space models (SSMs) for multimodal learning to address Transformer-scale computation.
• Propose VL-Mamba by replacing Transformer-based backbones with Mamba LLM and adding a 2D vision selective scan based MMC.
• Investigate how different vision encoders, LLM variants, and MMC architectures affect multimodal performance.
• Demonstrate competitive results on standard multimodal benchmarks and provide ablations to understand component contributions.

提案手法

• Use a pre-trained Mamba LLM as the backbone language model instead of Transformer-based LLMs.
• Incorporate a Vision Transformer as the vision encoder to extract image patch features.
• Introduce MultiModal Connector (MMC) with Vision Selective Scan (VSS) to bridge 2D visual data with 1D sequential modeling.
• Explore two 2D scan mechanisms (Bidirectional-Scan and Cross-Scan) to capture visual context efficiently.
• Evaluate three MMC variants (MLP, VSS-MLP, VSS-L2) and two vision encoders (CLIP-ViT-L and SigLIP-SO) through extensive ablations.
• Perform experiments on eight multimodal benchmarks comparing VL-Mamba with state-of-the-art MLLMs.

実験結果

リサーチクエスチョン

• RQ1 Does replacing Transformer backbones with Mamba LLMs improve efficiency and scalability for multimodal tasks?
• RQ2 How effective is a 2D Vision Selective Scan based MMC at bridging non-causal visual data with causal state-space modeling?
• RQ3 What is the impact of different vision encoders, MMC architectures, and scan mechanisms on multimodal benchmarks?
• RQ4 Can VL-Mamba achieve competitive performance with smaller parameter counts and less pretraining data than some large MLLMs?

主な発見

手法	LLM	PT	IT	VQAv2	GQA	SQA I	VQAT	POPE	MME	MMBench	MMVet
BLIP-2	Vicuna-13B	129M	-	41.0	41.0	61.0	42.5	85.3	1293.8	–	22.4
MiniGPT-4	Vicuna-7B	5M	5K	-	32.2	-	-	-	581.7	23.0	-
InstructBLIP	Vicuna-7B	129M	1.2M	-	49.2	60.5	50.1	-	-	36	26.2
InstructBLIP	Vicuna-13B	129M	1.2M	-	49.5	63.1	50.7	78.9	1212.8	-	25.6
Shikra	Vicuna-13B	600K	5.5M	77.4	-	-	-	-	-	58.8	-
Otter	LLaMA-7B	-	-	-	-	-	-	-	1292.3	48.3	24.6
mPLUG-Owl	LLaMA-7B	2.1M	102K	-	-	-	-	-	967.3	49.4	-
IDEFICS-9B	LLaMA-7B	353M	1M	50.9	38.4	-	25.9	-	-	48.2	-
IDEFICS-80B	LLaMA-65B	353M	1M	60.0	45.2	-	30.9	-	-	54.5	-
Qwen-VL	Qwen-7B	1.4B	50M	78.8	59.3	67.1	63.8	-	-	38.2	-
Qwen-VL-Chat	Qwen-7B	1.4B	50M	78.2	57.5	68.2	61.5	-	1487.5	60.6	-
LLaVA-1.5	Vicuna-7B	558K	665K	78.5	62.0	66.8	58.2	85.9	1510.7	64.3	30.5
LLaVA-1.5	Vicuna-13B	558K	665K	80.0	63.3	71.6	61.3	85.9	1531.3	67.7	35.4
LLaVA-Phi	Phi-2-2.7B	558K	665K	71.4	-	68.4	48.6	85.0	1335.1	59.8	28.9
MobileVLM-3B	MobileLLaMA-2.7B	558K	665K	-	59.0	61.2	47.5	84.9	1288.9	59.6	-
VL-Mamba	Mamba LLM-2.8B	558K	665K	76.6	56.2	65.4	48.9	84.4	1369.6	57.0	32.6

• VL-Mamba achieves competitive performance against other small MLLMs of similar size and can outperform some larger models on select benchmarks.
• The VL-Mamba variant with SigLIP-SO as the vision encoder and Mamba-2.8B-Slimpj LLM shows strong overall performance in ablations.
• VSS-L2 MMC architecture and Bidirectional Scan (BSM) generally yield strong results across benchmarks.
• VL-Mamba demonstrates the feasibility of applying state space models to multimodal learning tasks with competitive results.
• Ablations indicate that language model variant, vision encoder, MMC design, and scan mechanism all meaningfully affect performance.