[Paper Review] Fusion-Mamba for Cross-modality Object Detection

Wenhao Dong, Haodong Zhu+7 more|arXiv (Cornell University)|Apr 14, 2024

Advanced Image and Video Retrieval Techniques15 citations

TL;DR

This paper proposes Fusion-Mamba blocks to fuse RGB and IR features in a hidden state space using SSCS and DSSF, achieving state-of-the-art cross-modality object detection on multiple datasets.

ABSTRACT

Cross-modality fusing complementary information from different modalities effectively improves object detection performance, making it more useful and robust for a wider range of applications. Existing fusion strategies combine different types of images or merge different backbone features through elaborated neural network modules. However, these methods neglect that modality disparities affect cross-modality fusion performance, as different modalities with different camera focal lengths, placements, and angles are hardly fused. In this paper, we investigate cross-modality fusion by associating cross-modal features in a hidden state space based on an improved Mamba with a gating mechanism. We design a Fusion-Mamba block (FMB) to map cross-modal features into a hidden state space for interaction, thereby reducing disparities between cross-modal features and enhancing the representation consistency of fused features. FMB contains two modules: the State Space Channel Swapping (SSCS) module facilitates shallow feature fusion, and the Dual State Space Fusion (DSSF) enables deep fusion in a hidden state space. Through extensive experiments on public datasets, our proposed approach outperforms the state-of-the-art methods on $m$AP with 5.9% on $M^3FD$ and 4.9% on FLIR-Aligned datasets, demonstrating superior object detection performance. To the best of our knowledge, this is the first work to explore the potential of Mamba for cross-modal fusion and establish a new baseline for cross-modality object detection.

Motivation & Objective

• Motivate robust cross-modality object detection by addressing modality disparities between RGB and IR data.
• Develop a hidden-state interaction mechanism to better fuse cross-modality features.
• Design an efficient fusion module that improves representation coherence without excessive compute.
• Demonstrate state-of-the-art performance on public RGB-IR datasets across multiple backbones.

Proposed method

• Introduce Fusion-Mamba blocks (FMB) consisting of two modules: State Space Channel Swapping (SSCS) for shallow fusion and Dual State Space Fusion (DSSF) for deep fusion in a hidden state space.
• Map cross-modal features into a hidden state space to enable interaction and reduce modality disparities.
• SSCS performs channel swapping and applies a Visual State Space (VSS) block for shallow cross-modal interaction.
• DSSF projects features into a hidden state space with a gating mechanism to enable deep cross-modal fusion, followed by projection back and residual connections.
• Fuse enhanced RGB and IR features through addition to form fused features used by the YOLOv5/YOLOv8 backbones in the detection head.

Experimental results

Research questions

• RQ1 How can cross-modality disparities between RGB and IR features be mitigated during fusion to improve object detection accuracy?
• RQ2 Can a hidden-state space fusion approach outperform spatially-focused or transformer-based fusion for RGB-IR data?
• RQ3 What are the trade-offs in accuracy and computation when using Fusion-Mamba compared to transformer-based fusion methods?

Key findings

• Fusion-Mamba with RGB-IR data achieves state-of-the-art mAP improvements on M3FD (5.9% gain) and FLIR-Aligned (4.9% gain) over prior methods.
• On LLVIP, Fusion-Mamba with YOLOv5 backbone attains 62.8 mAP and 96.8 mAP50, while with YOLOv8 backbone it reaches 64.3 mAP and 97.0 mAP50.
• Fusion-Mamba with YOLOv8 consistently outperforms competitive baselines across backbones on LLVIP, M3FD, and FLIR-Aligned datasets.
• Ablation studies show that removing SSCS or DSSF reduces mAP by notable margins, demonstrating the necessity of both modules for effective cross-modality fusion.
• Compared to transformer-based fusion, Fusion-Mamba offers lower time complexity with noticeable speedups (e.g., 7–19 ms faster inference under the same setup).