[Paper Review] CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Jiaming Zhang, Huayao Liu+4 more|arXiv (Cornell University)|Mar 9, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes CMX, a unified transformer-based cross-modal fusion framework for RGB-X semantic segmentation that generalizes across diverse modalities like depth, thermal, polarization, event, and LiDAR. By introducing a Cross-Modal Feature Rectification Module (CM-FRM) for bidirectional feature calibration and a Feature Fusion Module (FFM) with cross-attention for long-range context exchange, CMX achieves state-of-the-art performance on nine benchmarks, including new SOTA on RGB-Event segmentation using the EventScape dataset.

ABSTRACT

Scene understanding based on image segmentation is a crucial component of autonomous vehicles. Pixel-wise semantic segmentation of RGB images can be advanced by exploiting complementary features from the supplementary modality (X-modality). However, covering a wide variety of sensors with a modality-agnostic model remains an unresolved problem due to variations in sensor characteristics among different modalities. Unlike previous modality-specific methods, in this work, we propose a unified fusion framework, CMX, for RGB-X semantic segmentation. To generalize well across different modalities, that often include supplements as well as uncertainties, a unified cross-modal interaction is crucial for modality fusion. Specifically, we design a Cross-Modal Feature Rectification Module (CM-FRM) to calibrate bi-modal features by leveraging the features from one modality to rectify the features of the other modality. With rectified feature pairs, we deploy a Feature Fusion Module (FFM) to perform sufficient exchange of long-range contexts before mixing. To verify CMX, for the first time, we unify five modalities complementary to RGB, i.e., depth, thermal, polarization, event, and LiDAR. Extensive experiments show that CMX generalizes well to diverse multi-modal fusion, achieving state-of-the-art performances on five RGB-Depth benchmarks, as well as RGB-Thermal, RGB-Polarization, and RGB-LiDAR datasets. Besides, to investigate the generalizability to dense-sparse data fusion, we establish an RGB-Event semantic segmentation benchmark based on the EventScape dataset, on which CMX sets the new state-of-the-art. The source code of CMX is publicly available at https://github.com/huaaaliu/RGBX_Semantic_Segmentation.

Motivation & Objective

• To address the lack of modality-agnostic models for multi-modal semantic segmentation in autonomous driving.
• To enable effective fusion of diverse, complementary sensing modalities (e.g., depth, thermal, event, LiDAR) with RGB in a single unified architecture.
• To improve robustness and accuracy in low-texture, low-light, and dynamic scenarios through cross-modal feature calibration.
• To establish a new benchmark for RGB-Event semantic segmentation to evaluate dense-sparse fusion capabilities.
• To develop a generalizable, efficient, and accurate solution for real-world autonomous vehicle perception.

Proposed method

• Proposes a Cross-Modal Feature Rectification Module (CM-FRM) that performs channel- and spatial-wise feature calibration using cross-attention between RGB and X-modality features.
• Employs a Feature Fusion Module (FFM) with sequence-to-sequence cross-attention and mixed channel embedding to enable bidirectional exchange of long-range contextual features.
• Uses Vision Transformer-based encoders (e.g., SegFormer) for joint feature extraction from RGB and X-modality inputs.
• Applies modality-agnostic fusion by processing all modalities through the same CMX architecture without architectural re-design per modality pair.
• Introduces a new RGB-Event semantic segmentation benchmark based on the EventScape dataset to evaluate dense-sparse fusion.
• Trains end-to-end with cross-entropy and Dice loss, optimizing for mIoU on multi-modal segmentation tasks.

Experimental results

Research questions

• RQ1 Can a single, unified model effectively fuse RGB with diverse, non-identical modalities such as depth, thermal, polarization, event, and LiDAR?
• RQ2 How can cross-modal feature calibration improve performance in scenarios with low texture, low light, or high dynamic range?
• RQ3 Does bidirectional cross-attention with feature rectification lead to better long-range context modeling than unidirectional or early fusion?
• RQ4 Can CMX generalize to sparse-dense data fusion, such as combining high-rate event data with standard RGB images?
• RQ5 What is the performance gain of CMX over existing modality-specific models across multiple benchmark datasets?

Key findings

• CMX achieves state-of-the-art mIoU of 56.8% on the NYU Depth V2 dataset using SegFormer-B5, outperforming prior methods.
• On the new RGB-Event benchmark based on EventScape, CMX sets a new SOTA mIoU, demonstrating strong generalization to sparse-dense fusion.
• CMX improves segmentation accuracy in low-light conditions by 5.2% mIoU over RGB-only baselines on the RGB-Thermal task.
• The CM-FRM module reduces noise sensitivity in LiDidar data, enabling complete and accurate segmentation of vehicles despite sparse or noisy point clouds.
• CMX with SegFormer-B2 achieves 54.1% mIoU with only 67.6 GFLOPs, offering a strong accuracy-efficiency trade-off for real-time applications.
• Qualitative results show CMX correctly segments challenging cases such as glass surfaces, moving objects, and low-texture regions (e.g., beds as sofas) using modality-specific cues.