[Paper Review] Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation

Junde Wu, Ji, Wei+5 more|arXiv (Cornell University)|Apr 25, 2023

Radiomics and Machine Learning in Medical Imaging222 citations

TL;DR

This paper introduces Med-SA, a parameter-efficient adaptation of SAM for medical image segmentation, using SD-Trans for 2D-to-3D and HyP-Adpt for prompt-conditioned adaptation, achieving state-of-the-art results across 17 tasks while updating only about 2% of parameters.

ABSTRACT

The Segment Anything Model (SAM) has recently gained popularity in the field of image segmentation due to its impressive capabilities in various segmentation tasks and its prompt-based interface. However, recent studies and individual experiments have shown that SAM underperforms in medical image segmentation, since the lack of the medical specific knowledge. This raises the question of how to enhance SAM's segmentation capability for medical images. In this paper, instead of fine-tuning the SAM model, we propose the Medical SAM Adapter (Med-SA), which incorporates domain-specific medical knowledge into the segmentation model using a light yet effective adaptation technique. In Med-SA, we propose Space-Depth Transpose (SD-Trans) to adapt 2D SAM to 3D medical images and Hyper-Prompting Adapter (HyP-Adpt) to achieve prompt-conditioned adaptation. We conduct comprehensive evaluation experiments on 17 medical image segmentation tasks across various image modalities. Med-SA outperforms several state-of-the-art (SOTA) medical image segmentation methods, while updating only 2\% of the parameters. Our code is released at https://github.com/KidsWithTokens/Medical-SAM-Adapter.

Motivation & Objective

• Motivate improving SAM’s performance on medical images by incorporating medical-domain knowledge without full fine-tuning.
• Develop a lightweight adaptation framework (Med-SA) with (i) SD-Trans to enable 2D-to-3D adaptation, and (ii) HyP-Adpt for prompt-conditioned adaptation.
• Evaluate Med-SA across 17 medical segmentation tasks spanning multiple modalities and compare against SAM, MedSAM, and SOTA methods.
• Demonstrate that substantial gains are achievable by updating only ~2% of SAM’s parameters.

Proposed method

• Incorporate Adaption-based parameter-efficient fine-tuning by inserting Adapter modules into both the SAM encoder and decoder.
• Introduce Space-Depth Transpose (SD-Trans) to capture 3D contextual information by processing spatial and depth dimensions via split attention branches.
• Propose Hyper-Prompting Adapter (HyP-Adpt) to generate prompt-conditioned adapter weights from visual prompts, enabling wide and deep prompt-adaptation with few additional parameters.
• Process 2D medical images with standard SAM adapters; extend to 3D volumes using SD-Trans; fuse prompt information through HyP-Adpt in the decoder.
• Train with interactive prompts (points and bounding boxes) using random and iterative sampling to mimic user interactions.

Experimental results

Research questions

• RQ1 Can a lightweight, adapter-based approach adapt SAM to the medical domain without full fine-tuning?
• RQ2 Does SD-Trans enable effective 2D-to-3D segmentation within SAM’s framework?
• RQ3 Can HyP-Adpt provide prompt-conditioned adaptation with minimal parameter overhead?
• RQ4 How does Med-SA perform across diverse medical imaging modalities and tasks compared to SAM, MedSAM, and specialized SOTA methods?

Key findings

• Med-SA achieves state-of-the-art performance across 17 medical segmentation tasks, outperforming SAM and fully fine-tuned MedSAM while updating only about 2% of SAM’s parameters.
• On BTCV abdominal multi-organ segmentation, Med-SA with 1-point prompts already surpasses prior methods, and with BBox 0.75 reaches Dice up to 89.8%, beating Swin-UNetr by 2.9%.
• Med-SA outperforms MedSAM and several SOTA domain-specific methods (e.g., nnUNet, TransUNet, UNetr, Swin-UNetr) across multiple modalities (CT, MRI, ultrasound, fundus, dermoscopic).
• SD-Trans and HyP-Adpt ablations show 2D-to-3D design significantly boosts performance on BTCV and BrainTumor benchmarks, and HyP-Adpt provides substantial gains over simple addition/concatenation of prompt embeddings.
• Med-SA demonstrates strong generalization to multi-modality datasets (REFUGE2, BraTs, TNMIX, ISIC) with consistent gains across tasks.