[Paper Review] MambaDFuse: A Mamba-based Dual-phase Model for Multi-modality Image Fusion

Zhe Li, Haiwei Pan+3 more|arXiv (Cornell University)|Apr 12, 2024

Advanced Image Fusion Techniques17 citations

TL;DR

MambaDFuse introduces a Mamba-based dual-phase framework with a dual-level feature extractor and dual-phase fusion for MMIF, achieving state-of-the-art results on IVF and MIF and improving downstream object detection.

ABSTRACT

Multi-modality image fusion (MMIF) aims to integrate complementary information from different modalities into a single fused image to represent the imaging scene and facilitate downstream visual tasks comprehensively. In recent years, significant progress has been made in MMIF tasks due to advances in deep neural networks. However, existing methods cannot effectively and efficiently extract modality-specific and modality-fused features constrained by the inherent local reductive bias (CNN) or quadratic computational complexity (Transformers). To overcome this issue, we propose a Mamba-based Dual-phase Fusion (MambaDFuse) model. Firstly, a dual-level feature extractor is designed to capture long-range features from single-modality images by extracting low and high-level features from CNN and Mamba blocks. Then, a dual-phase feature fusion module is proposed to obtain fusion features that combine complementary information from different modalities. It uses the channel exchange method for shallow fusion and the enhanced Multi-modal Mamba (M3) blocks for deep fusion. Finally, the fused image reconstruction module utilizes the inverse transformation of the feature extraction to generate the fused result. Through extensive experiments, our approach achieves promising fusion results in infrared-visible image fusion and medical image fusion. Additionally, in a unified benchmark, MambaDFuse has also demonstrated improved performance in downstream tasks such as object detection. Code with checkpoints will be available after the peer-review process.

Motivation & Objective

• Motivate MMIF to balance fusion quality and computational efficiency.
• Propose a Mamba-based backbone to overcome CNN/Transformer limitations in MMIF.
• Design dual-level feature extraction to capture both local and long-range modality-specific information.
• Develop a dual-phase fusion mechanism to integrate global overview and local detail from multiple modalities.
• Demonstrate improvements on IVF (Infrared-Visible) and MIF (medical) fusion tasks and downstream detection.

Proposed method

• Use a dual-level feature extractor combining CNNs for low-level features and Mamba blocks for high-level long-range features.
• Implement a shallow fusion module with channel exchange to rapidly fuse global information.
• Develop a deep fusion module using Multi-modal Mamba (M3) blocks to guide modality-fused features with cross-modal information.
• Reconstruct the fused image via inverse transformation of the feature extraction pipeline.
• Train with a loss combining SSIM, texture, and intensity terms as in prior SwinFusion work.

Experimental results

Research questions

• RQ1 Can Mamba-based architectures achieve efficient and effective MMIF compared to CNN- or Transformer-based backbones?
• RQ2 Does a dual-level feature extractor improve modality-specific feature capture for MMIF?
• RQ3 Can dual-phase fusion (shallow channel exchange plus deep M3-based fusion) produce superior fusion features for IVF and MIF?
• RQ4 Do fused images produced by MambaDFuse enhance downstream tasks such as object detection?

Key findings

• MambaDFuse achieves leading performance across IVF and MIF benchmarks on multiple datasets (IVF: MSRS, RoadScene, M3FD; MIF: MRI-CT, MRI-PET, MRI-SPECT).
• The shallow fusion stage via channel exchange effectively integrates cross-modality information without extra parameters.
• The deep fusion stage with M3 blocks improves detail-oriented fusion guided by modality-specific features.
• Fused images show improved metrics (e.g., MI, VIF, SSIM, Qabf) and clearer object delineation in qualitative comparisons.
• Unified benchmarking indicates improved downstream object detection performance when using MambaDFuse fused images.