[Paper Review] Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts

Chenyang Zhu, Bin Xiao+3 more|arXiv (Cornell University)|Dec 5, 2024

Big Data and Business IntelligenceBusiness, Management and Accounting3 citations

TL;DR

This paper proposes MLE-SAM, a Mixture of LoRA Experts framework that adapts the Segment Anything Model (SAM) for multi-modal semantic segmentation by training modality-specific LoRA adapters while freezing SAM’s original weights. It achieves state-of-the-art performance, especially under missing modality conditions, with a 32.15% mIoU improvement over prior methods on the MUSES dataset.

ABSTRACT

The recent Segment Anything Model (SAM) represents a significant breakthrough in scaling segmentation models, delivering strong performance across various downstream applications in the RGB modality. However, directly applying SAM to emerging visual modalities, such as depth and event data results in suboptimal performance in multi-modal segmentation tasks. In this paper, we make the first attempt to adapt SAM for multi-modal semantic segmentation by proposing a Mixture of Low-Rank Adaptation Experts (MoE-LoRA) tailored for different input visual modalities. By training only the MoE-LoRA layers while keeping SAM's weights frozen, SAM's strong generalization and segmentation capabilities can be preserved for downstream tasks. Specifically, to address cross-modal inconsistencies, we propose a novel MoE routing strategy that adaptively generates weighted features across modalities, enhancing multi-modal feature integration. Additionally, we incorporate multi-scale feature extraction and fusion by adapting SAM's segmentation head and introducing an auxiliary segmentation head to combine multi-scale features for improved segmentation performance effectively. Extensive experiments were conducted on three multi-modal benchmarks: DELIVER, MUSES, and MCubeS. The results consistently demonstrate that the proposed method significantly outperforms state-of-the-art approaches across diverse scenarios. Notably, under the particularly challenging condition of missing modalities, our approach exhibits a substantial performance gain, achieving an improvement of 32.15% compared to existing methods.

Motivation & Objective

• Address the challenge of applying the pre-trained Segment Anything Model (SAM) to non-RGB visual modalities like depth and event data, which exhibit distinct characteristics and degrade SAM’s performance.
• Overcome cross-modal inconsistencies and suboptimal feature integration when fusing heterogeneous modalities such as RGB, depth, event, and LiDARD.
• Preserve SAM’s strong generalization and zero-shot capabilities by fine-tuning only lightweight LoRA adapters while keeping the original model weights frozen.
• Enhance robustness in real-world scenarios with missing or noisy modalities through a dynamic MoE routing mechanism and multi-scale feature fusion.
• Achieve superior segmentation accuracy across diverse multi-modal benchmarks, including DELIVER, MUSES, and MCubeS, under both standard and adverse conditions.

Proposed method

• Introduce a Mixture of Low-Rank Adaptation Experts (MoE-LoRA) framework, where each modality (e.g., RGB, depth, event) is assigned a dedicated LoRA adapter for modality-specific adaptation.
• Design a dynamic MoE routing mechanism that adaptively computes attention weights across modalities to generate fused, weighted features, improving cross-modal consistency and integration.
• Modify SAM’s segmentation head by adding an auxiliary head to extract and fuse multi-scale features, enhancing feature representation and segmentation accuracy.
• Train only the LoRA adapters and auxiliary head while freezing all original SAM weights, ensuring preservation of SAM’s generalization capacity.
• Implement a dual-pathway mask prediction strategy that leverages both the original and auxiliary heads to improve mask quality through multi-scale fusion.
• Apply the MoE-LoRA framework to three benchmarks: DELIVER, MUSES, and MCubeS, with extensive ablation and robustness evaluations under noise and missing modality conditions.

Experimental results

Research questions

• RQ1 Can LoRA-based fine-tuning effectively adapt the pre-trained Segment Anything Model to non-RGB visual modalities without fine-tuning the original weights?
• RQ2 How can a Mixture of Experts routing mechanism improve multi-modal feature fusion and handle cross-modal inconsistencies in heterogeneous data?
• RQ3 To what extent does multi-scale feature fusion via an auxiliary segmentation head enhance segmentation performance in multi-modal settings?
• RQ4 How does the proposed MLE-SAM framework perform under real-world challenges such as missing or noisy modalities compared to existing state-of-the-art methods?
• RQ5 What is the contribution of each component (MoE routing, multi-scale fusion, auxiliary head) to the overall performance gain in multi-modal semantic segmentation?

Key findings

• MLE-SAM achieves a +4.9% mIoU improvement over state-of-the-art methods on the DELIVER dataset with four modalities (R-D-E-L).
• On the MUSES dataset with three modalities (F-E-L), MLE-SAM achieves a +28.14% mIoU gain, demonstrating strong performance in complex multi-modal scenarios.
• Under missing modality conditions, MLE-SAM achieves a 32.15% mIoU improvement over existing methods on the MUSES dataset, highlighting its robustness.
• In noisy testing conditions, MLE-SAM outperforms CWSAM and SAM-LoRA across all modalities, with a 32.42% mIoU gain on RGB under random noise.
• The MoE routing mechanism effectively balances contributions from dense (RGB, depth) and sparse (event, LiDAR) modalities, with event and LiDAR maintaining high mIoU values under both Gaussian and random noise.
• Ablation studies confirm that the auxiliary head and MoE routing mechanism each make significant, independent contributions to the final performance gain.