[Paper Review] FusionMamba: Dynamic Feature Enhancement for Multimodal Image Fusion with Mamba

Xinyu Xie, Yawen Cui+3 more|arXiv (Cornell University)|Apr 15, 2024

Advanced Image Fusion Techniques4 citations

TL;DR

FusionMamba proposes a novel multimodal image fusion framework leveraging an improved Mamba-based architecture with dynamic feature enhancement to balance global context modeling and local detail preservation. By integrating dynamic convolution, channel attention, and a cross-modality fusion module, it achieves state-of-the-art performance across diverse datasets including medical (CT-MRI, PET-MRI), infrared-visible, and biomedical (GFP-PC) fusion tasks, with significant gains in key metrics and reduced computational cost.

ABSTRACT

Multimodal image fusion aims to integrate information from different imaging techniques to produce a comprehensive, detail-rich single image for downstream vision tasks. Existing methods based on local convolutional neural networks (CNNs) struggle to capture global features efficiently, while Transformer-based models are computationally expensive, although they excel at global modeling. Mamba addresses these limitations by leveraging selective structured state space models (S4) to effectively handle long-range dependencies while maintaining linear complexity. In this paper, we propose FusionMamba, a novel dynamic feature enhancement framework that aims to overcome the challenges faced by CNNs and Vision Transformers (ViTs) in computer vision tasks. The framework improves the visual state-space model Mamba by integrating dynamic convolution and channel attention mechanisms, which not only retains its powerful global feature modeling capability, but also greatly reduces redundancy and enhances the expressiveness of local features. In addition, we have developed a new module called the dynamic feature fusion module (DFFM). It combines the dynamic feature enhancement module (DFEM) for texture enhancement and disparity perception with the cross-modal fusion Mamba module (CMFM), which focuses on enhancing the inter-modal correlation while suppressing redundant information. Experiments show that FusionMamba achieves state-of-the-art performance in a variety of multimodal image fusion tasks as well as downstream experiments, demonstrating its broad applicability and superiority.

Motivation & Objective

• To address the limitations of CNNs in capturing global context and Transformers in computational complexity for multimodal image fusion.
• To overcome the static nature of convolutional operations that fail to adapt to spatial variations in multimodal inputs.
• To develop a lightweight, efficient architecture that enhances inter-modal correlation while suppressing redundant information.
• To improve fusion performance by integrating dynamic feature enhancement with selective state space modeling for better texture and difference perception.
• To validate the generalization capability of the proposed method across diverse multimodal image fusion benchmarks.

Proposed method

• Proposes an improved efficient Mamba model by integrating dynamic convolution and channel attention to enhance local feature extraction and reduce channel redundancy.
• Introduces a Dynamic Feature Fusion Module (DFFM) comprising two Dynamic Feature Enhancement Modules (DFEM) for adaptive texture and difference perception.
• Employs a Cross-Modality Fusion Mamba Module (CMFM) to model long-range dependencies and enhance correlation between modalities.
• Utilizes a hybrid loss function combining SSIM, texture, and intensity losses to preserve structural integrity, edge details, and brightness consistency.
• Adopts a selective structured state space model with linear complexity to efficiently model long-range dependencies without the quadratic cost of self-attention.
• Employs a parallel scanning algorithm and structured reparameterization to accelerate inference and maintain high performance.

Experimental results

Research questions

• RQ1 Can a Mamba-based architecture effectively model long-range dependencies in multimodal image fusion with linear complexity?
• RQ2 How does dynamic convolution and channel attention improve local feature enhancement in multimodal fusion?
• RQ3 To what extent does the proposed DFFM module enhance inter-modal correlation while suppressing redundant information?
• RQ4 Does the integration of CMFM lead to better fusion performance compared to static or attention-based fusion modules?
• RQ5 Can FusionMamba generalize across diverse multimodal datasets including medical, infrared-visible, and biomedical imaging?

Key findings

• FusionMamba achieves state-of-the-art performance on CT-MRI, PET-MRI, SPECT-MRI, IR-VIS, and GFP-PC datasets, with a VIF score of 0.7717 and SF of 17.0171 on the IR-VIS benchmark.
• The model reduces FLOPs to 16.50G, significantly lower than previous SOTA methods, demonstrating high computational efficiency.
• Ablation studies confirm that removing the CMFM or DFEM modules leads to a drop in MS-SSIM (by 4.36%) and SF (by 3.2%), proving their critical role.
• The full loss function (SSIM + texture + intensity) yields the highest MS-SSIM (0.9331) and SF (17.0171), outperforming ablated versions.
• Replacing the Mamba backbone with a Transformer leads to a 1.5% drop in VIF and 1.5% in SF, confirming Mamba’s superiority in efficiency and performance.
• The model achieves a 13.5% improvement in Q^{AB/F} over U2Fusion and a 10.2% improvement over the next SOTA method on the GFP-PC dataset.