[Paper Review] A Unified Framework with Multimodal Fine-tuning for Remote Sensing Semantic Segmentation

Xianping Ma, Xiaokang Zhang+2 more|arXiv (Cornell University)|Oct 15, 2024

Geographic Information Systems Studies4 citations

TL;DR

This paper proposes MANet, a unified framework that fine-tunes the Segment Anything Model (SAM) for multimodal remote sensing semantic segmentation using a Multimodal Adapter (MMAdapter) and a pyramid-based Deep Fusion Module (DFM). By leveraging SAM’s general knowledge through parameter-efficient adaptation, MANet achieves state-of-the-art performance on ISPRS Vaihingen and Potsdam datasets, demonstrating for the first time that SAM can effectively handle non-optical DSM data in segmentation tasks.

ABSTRACT

Multimodal remote sensing data, acquired from diverse sensors, offer a comprehensive and integrated perspective of the Earth's surface. Leveraging multimodal fusion techniques, semantic segmentation enables detailed and accurate analysis of geographic scenes, surpassing single-modality approaches. Building on advancements in vision foundation models, particularly the Segment Anything Model (SAM), this study proposes a unified framework incorporating a novel Multimodal Fine-tuning Network (MFNet) for remote sensing semantic segmentation. The proposed framework is designed to seamlessly integrate with various fine-tuning mechanisms, demonstrated through the inclusion of Adapter and Low-Rank Adaptation (LoRA) as representative examples. This extensibility ensures the framework's adaptability to other emerging fine-tuning strategies, allowing models to retain SAM's general knowledge while effectively leveraging multimodal data. Additionally, a pyramid-based Deep Fusion Module (DFM) is introduced to integrate high-level geographic features across multiple scales, enhancing feature representation prior to decoding. This work also highlights SAM's robust generalization capabilities with Digital Surface Model (DSM) data, a novel application. Extensive experiments on three benchmark multimodal remote sensing datasets, ISPRS Vaihingen, ISPRS Potsdam and MMHunan, demonstrate that the proposed MFNet significantly outperforms existing methods in multimodal semantic segmentation, setting a new standard in the field while offering a versatile foundation for future research and applications. The source code for this work is accessible at https://github.com/sstary/SSRS.

Motivation & Objective

• Address the challenge of applying large vision foundation models like SAM to multimodal remote sensing data, which differ significantly from natural images in sensor type, resolution, and spectral characteristics.
• Overcome the limitations of existing models that are trained only on task-specific data by leveraging the general knowledge embedded in foundation models such as SAM.
• Enable effective fusion of multimodal remote sensing data—particularly optical and DSM—by introducing a novel Multimodal Adapter (MMAdapter) that fine-tunes SAM’s image encoder without full fine-tuning.
• Demonstrate the feasibility and effectiveness of using SAM’s image encoder for non-optical data such as Digital Surface Models (DSM), extending its applicability beyond RGB images.
• Develop a scalable, parameter-efficient framework suitable for deployment on single-GPU hardware, enabling broader adoption of large foundation models in resource-constrained remote sensing applications.

Proposed method

• Introduce a Multimodal Adapter (MMAdapter) that is inserted into the ViT-based image encoder of SAM to enable parameter-efficient fine-tuning for multimodal remote sensing data.
• The MMAdapter enables continuous fusion of multimodal features (e.g., optical and DSM) at multiple encoder blocks, preserving task-agnostic knowledge while adapting to remote sensing modality distributions.
• Integrate a pyramid-based Deep Fusion Module (DFM) that aggregates high-level, multi-scale features from different modalities before decoding, enhancing semantic representation learning.
• Use a prompt encoder and mask decoder from SAM unchanged, leveraging its strong zero-shot generalization capability for segmentation head inference.
• Apply adapter-based fine-tuning with frozen main weights, minimizing trainable parameters and memory footprint while maintaining high performance.
• Train the model end-to-end on fine-resolution multimodal datasets (ISPRS Vaihingen and Potsdam), with loss functions optimized for segmentation accuracy.

Experimental results

Research questions

• RQ1 Can the Segment Anything Model (SAM), pre-trained on natural images, be effectively adapted to multimodal remote sensing semantic segmentation tasks involving non-optical data like DSM?
• RQ2 How effective is the proposed Multimodal Adapter (MMAdapter) in enabling parameter-efficient fine-tuning of SAM’s image encoder for remote sensing data without full fine-tuning?
• RQ3 What is the contribution of the pyramid-based Deep Fusion Module (DFM) in enhancing feature integration across multiple scales and modalities in remote sensing segmentation?
• RQ4 To what extent does the general knowledge from SAM transfer to remote sensing scenes, particularly when using non-optical modalities such as DSM?
• RQ5 Can the proposed MANet framework achieve state-of-the-art performance while maintaining low computational and memory costs on a single GPU?

Key findings

• MANet achieves a mean Intersection over Union (mIoU) of 85.03% on the ISPRS Potsdam dataset using the ViT-H backbone, significantly outperforming existing methods such as FTransUNet (84.23%) and CMGFNet (82.26%).
• The ablation study confirms that both the MMAdapter and DFM are essential: removing either reduces mIoU by over 0.5 points, with the full model achieving 84.72% mIoU on Vaihingen.
• The model maintains high performance even with reduced batch size (from 10 to 6) on the ViT-H backbone, indicating robustness and efficiency under hardware constraints.
• The use of a single NVIDIA RTX 3090 GPU enables training of the ViT-L and ViT-H backbones, with only 56.67M and 111.28M trainable parameters respectively, demonstrating low memory and parameter overhead.
• Heatmap visualization confirms that the fine-tuned SAM encoder learns to effectively utilize DSM data, showing that SAM can generalize to non-optical modalities with proper adaptation.
• This work is the first to validate SAM’s effectiveness on DSM data, proving that foundation models can be successfully adapted to multimodal remote sensing tasks with minimal architectural changes.