[Paper Review] Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning

Shaohua Dong, Yunhe Feng+4 more|arXiv (Cornell University)|Dec 1, 2023

Visual Attention and Saliency Detection4 citations

TL;DR

This paper proposes DPLNet, a training-efficient multimodal semantic segmentation framework that adapts a frozen pre-trained RGB model using dual prompt learning. By introducing a lightweight multimodal prompt generator (MPG) and multimodal feature adapter (MFA), DPLNet achieves state-of-the-art performance on four RGB-D/T benchmarks with only 3.88M trainable parameters (4.4% of the backbone), significantly reducing training cost and deployment burden.

ABSTRACT

Multimodal (e.g., RGB-Depth/RGB-Thermal) fusion has shown great potential for improving semantic segmentation in complex scenes (e.g., indoor/low-light conditions). Existing approaches often fully fine-tune a dual-branch encoder-decoder framework with a complicated feature fusion strategy for achieving multimodal semantic segmentation, which is training-costly due to the massive parameter updates in feature extraction and fusion. To address this issue, we propose a surprisingly simple yet effective dual-prompt learning network (dubbed DPLNet) for training-efficient multimodal (e.g., RGB-D/T) semantic segmentation. The core of DPLNet is to directly adapt a frozen pre-trained RGB model to multimodal semantic segmentation, reducing parameter updates. For this purpose, we present two prompt learning modules, comprising multimodal prompt generator (MPG) and multimodal feature adapter (MFA). MPG works to fuse the features from different modalities in a compact manner and is inserted from shadow to deep stages to generate the multi-level multimodal prompts that are injected into the frozen backbone, while MPG adapts prompted multimodal features in the frozen backbone for better multimodal semantic segmentation. Since both the MPG and MFA are lightweight, only a few trainable parameters (3.88M, 4.4% of the pre-trained backbone parameters) are introduced for multimodal feature fusion and learning. Using a simple decoder (3.27M parameters), DPLNet achieves new state-of-the-art performance or is on a par with other complex approaches on four RGB-D/T semantic segmentation datasets while satisfying parameter efficiency. Moreover, we show that DPLNet is general and applicable to other multimodal tasks such as salient object detection and video semantic segmentation. Without special design, DPLNet outperforms many complicated models. Our code will be available at github.com/ShaohuaDong2021/DPLNet.

Motivation & Objective

• Address the high training cost and parameter inefficiency of existing dual-branch encoder-decoder models for multimodal semantic segmentation.
• Reduce the number of trainable parameters in multimodal feature fusion and adaptation while maintaining or improving performance.
• Enable deployment-friendly models by freezing the pre-trained RGB backbone and introducing only lightweight modules.
• Generalize the proposed method to other multimodal dense prediction tasks such as salient object detection and video semantic segmentation.
• Overcome domain gap issues between RGB and auxiliary modalities (e.g., depth, thermal) through effective prompt-based adaptation without full fine-tuning.

Proposed method

• Introduce a multimodal prompt generator (MPG) that fuses features from different modalities (e.g., RGB and depth/thermal) at multiple stages of a frozen backbone, generating multi-level multimodal prompts.
• Insert MPG modules progressively from shallow to deep stages to inject modality-complementary features into the frozen encoder.
• Design a multimodal feature adapter (MFA) that uses learnable tokens and cross-attention to adapt the frozen backbone to multimodal inputs, enhancing feature representation.
• Apply MFA at each stage to refine the fused multimodal features, enabling effective interaction between modality-specific features and the frozen network.
• Use a simple, lightweight decoder (3.27M parameters) to produce segmentation masks, minimizing additional parameter overhead.
• Train only the MPG and MFA modules (3.88M parameters) while keeping the pre-trained RGB backbone frozen, achieving parameter efficiency.

Experimental results

Research questions

• RQ1 Can a frozen pre-trained RGB model be effectively adapted for multimodal semantic segmentation with minimal parameter updates?
• RQ2 Does a dual-prompt learning framework—featuring a multimodal prompt generator and a feature adapter—outperform complex, fully fine-tuned dual-branch architectures in both accuracy and efficiency?
• RQ3 How does the placement and design of MPG and MFA modules affect performance across different semantic segmentation benchmarks?
• RQ4 Can the proposed method generalize beyond semantic segmentation to other multimodal dense prediction tasks like salient object detection and video segmentation?
• RQ5 To what extent does prompt-based adaptation mitigate the domain gap between RGB and auxiliary modalities (e.g., depth, thermal) without full fine-tuning?

Key findings

• DPLNet achieves state-of-the-art or competitive performance on four RGB-D/T semantic segmentation benchmarks (NYUDv2, NYUv2, S3DIS, and RGB-T SOD) with only 3.88M trainable parameters.
• On NYUDv2, DPLNet achieves 0.583 mIoU, outperforming a fully fine-tuned baseline (0.581 mIoU) while being significantly more parameter-efficient.
• Abllation studies confirm that both MPG and MFA are essential: removing either reduces mIoU by 0.9%, and optimal performance is achieved with 30 learnable tokens and 32-dimensional prompts.
• DPLNet generalizes effectively to other multimodal tasks: it achieves SOTA performance on five RGB-D salient object detection benchmarks and outperforms MVNet on RGB-T video semantic segmentation (0.579 mIoU vs. 0.545 mIoU) with 12× fewer trainable parameters.
• Despite using only a single-frame input, DPLNet achieves superior performance in video semantic segmentation compared to methods using temporal modeling, demonstrating strong generalization and efficiency.