[Paper Review] AsymFormer: Asymmetrical Cross-Modal Representation Learning for Mobile Platform Real-Time RGB-D Semantic Segmentation

Siqi Du, Weixi Wang+4 more|arXiv (Cornell University)|Sep 25, 2023

Advanced Neural Network ApplicationsComputer Science3 citations

TL;DR

AsymFormer proposes an asymmetrical cross-modal representation learning framework for real-time RGB-D semantic segmentation on mobile platforms, using a lightweight Transformer backbone for depth and a CNN for RGB to reduce redundancy. It achieves 52.0% mIoU on NYUv2 and 65 FPS inference speed (79 FPS with mixed precision), outperforming prior methods in speed-accuracy balance while minimizing parameter count through localized attention and cross-modal correlation modules.

ABSTRACT

Understanding indoor scenes is crucial for urban studies. Considering the dynamic nature of indoor environments, effective semantic segmentation requires both real-time operation and high accuracy.To address this, we propose AsymFormer, a novel network that improves real-time semantic segmentation accuracy using RGB-D multi-modal information without substantially increasing network complexity. AsymFormer uses an asymmetrical backbone for multimodal feature extraction, reducing redundant parameters by optimizing computational resource distribution. To fuse asymmetric multimodal features, a Local Attention-Guided Feature Selection (LAFS) module is used to selectively fuse features from different modalities by leveraging their dependencies. Subsequently, a Cross-Modal Attention-Guided Feature Correlation Embedding (CMA) module is introduced to further extract cross-modal representations. The AsymFormer demonstrates competitive results with 54.1% mIoU on NYUv2 and 49.1% mIoU on SUNRGBD. Notably, AsymFormer achieves an inference speed of 65 FPS (79 FPS after implementing mixed precision quantization) on RTX3090, demonstrating that AsymFormer can strike a balance between high accuracy and efficiency.

Motivation & Objective

• To address the inefficiency of symmetric dual-branch networks in RGB-D semantic segmentation, which double computational cost without proportional accuracy gains.
• To reduce model redundancy by designing an asymmetrical backbone with a lightweight depth branch and a powerful RGB branch.
• To improve feature fusion efficiency by introducing a Local Attention-Guided Feature Selection (LAFS) module that models spatial-channel attention in parallel.
• To enhance cross-modal representation learning with a Cross-Modal Attention (CMA) module that captures multi-modal self-similarity with minimal parameter increase.
• To achieve a superior speed-accuracy trade-off for deployment on mobile robotic platforms, prioritizing real-time inference without sacrificing segmentation quality.

Proposed method

• AsymFormer uses a CNN-based backbone for RGB features and a smaller, parameter-efficient Transformer for depth features to reduce computational redundancy.
• The Local Attention-Guided Feature Selection (LAFS) module computes spatial-channel attention weights in parallel using learnable channel weights and spatial attention, enabling fast, selective fusion.
• LAFS models spatial attention by estimating modality differences through learnable weights, improving feature selection while maintaining high inference speed.
• The Cross-Modal Attention (CMA) module enhances cross-modal representation by modeling self-similarity across modalities, improving feature correlation with minimal parameter overhead.
• Feature fusion is guided by attention maps that dynamically weight contributions from RGB and depth branches based on spatial and channel dependencies.
• Mixed-precision inference is applied to further accelerate inference, achieving 79 FPS on RTX 3090 with no accuracy drop.

Figure 1: Overview of AsymFormer.

Experimental results

Research questions

• RQ1 Can an asymmetrical backbone design reduce parameter redundancy in multi-modal semantic segmentation without sacrificing accuracy?
• RQ2 How can feature selection be optimized for real-time performance while maintaining spatial and channel-wise attention fidelity?
• RQ3 To what extent can multi-modal self-similarity features improve segmentation accuracy with minimal parameter increase?
• RQ4 Can a lightweight cross-attention mechanism effectively enhance cross-modal representation learning in real-time settings?
• RQ5 Does the proposed method achieve a better speed-accuracy trade-off than existing SOTA methods on mobile robotic platforms?

Key findings

• AsymFormer achieves 52.0% mIoU on the NYUv2 dataset, outperforming most existing methods in both accuracy and speed.
• On RTX 3090, AsymFormer attains 65 FPS inference speed with full precision and 79 FPS with mixed-precision quantization, significantly outperforming prior methods in speed.
• The model maintains 52.0% mIoU with only 33M parameters, demonstrating high efficiency and low parameter count.
• On the SUNRGBD dataset, AsymFormer achieves 49.1% mIoU, showing competitive performance despite lower-quality depth images in the dataset.
• Visualizations confirm that LAFS produces more coherent and informative attention maps than CBAM, with better edge preservation and object consistency.
• The ablation study confirms that the combination of asymmetrical backbone, LAFS, and CMA modules leads to significant gains in both speed and accuracy.

Figure 2: Difference between CBAM and LAFS’s spatial attention map.