[Paper Review] ProMISe: Promptable Medical Image Segmentation using SAM

Jinfeng Wang, Sifan Song+5 more|arXiv (Cornell University)|Mar 7, 2024

Brain Tumor Detection and ClassificationNeuroscience3 citations

TL;DR

This paper proposes ProMISe, a non-fine-tuned framework for promptable medical image segmentation using the Segment Anything Model (SAM). By introducing an Auto-Prompting Module (APM) for adaptive Euclidean prompt generation and Incremental Pattern Shifting (IPS) to non-invasively adapt SAM’s mask decoder, ProMISe achieves state-of-the-art performance across multiple medical imaging benchmarks while keeping all SAM parameters frozen, significantly reducing training costs and preserving prompting interpretability.

ABSTRACT

With the proposal of the Segment Anything Model (SAM), fine-tuning SAM for medical image segmentation (MIS) has become popular. However, due to the large size of the SAM model and the significant domain gap between natural and medical images, fine-tuning-based strategies are costly with potential risk of instability, feature damage and catastrophic forgetting. Furthermore, some methods of transferring SAM to a domain-specific MIS through fine-tuning strategies disable the model's prompting capability, severely limiting its utilization scenarios. In this paper, we propose an Auto-Prompting Module (APM), which provides SAM-based foundation model with Euclidean adaptive prompts in the target domain. Our experiments demonstrate that such adaptive prompts significantly improve SAM's non-fine-tuned performance in MIS. In addition, we propose a novel non-invasive method called Incremental Pattern Shifting (IPS) to adapt SAM to specific medical domains. Experimental results show that the IPS enables SAM to achieve state-of-the-art or competitive performance in MIS without the need for fine-tuning. By coupling these two methods, we propose ProMISe, an end-to-end non-fine-tuned framework for Promptable Medical Image Segmentation. Our experiments demonstrate that both using our methods individually or in combination achieves satisfactory performance in low-cost pattern shifting, with all of SAM's parameters frozen.

Motivation & Objective

• To address the high cost and instability of fine-tuning large foundation models like SAM for medical image segmentation (MIS).
• To preserve SAM’s interactive prompting capability while adapting it to medical domains with limited domain shift.
• To develop a lightweight, non-invasive method for domain adaptation that avoids catastrophic forgetting and feature damage.
• To enable end-to-end pattern shifting without fine-tuning, maintaining interpretability and efficiency in clinical deployment.

Proposed method

• The Auto-Prompting Module (APM) generates adaptive Euclidean prompts in the target domain using features from SAM’s image encoder, enabling improved zero-shot performance without fine-tuning.
• Incremental Pattern Shifting (IPS) introduces learnable tokens that shift the prior pattern knowledge in SAM’s mask decoder, adapting it to medical image characteristics.
• The Pattern Embedding (PaE) module identifies domain gaps between source (natural) and target (medical) images to guide effective pattern shifting.
• IPS is implemented by injecting learnable tokens into the mask decoder’s output tokens, allowing non-invasive adaptation without modifying SAM’s original parameters.
• The ProMISe framework combines APM and IPS for end-to-end, non-fine-tuned pattern shifting, maintaining full prompting capability.
• All experiments are conducted with all SAM parameters frozen, using only 1.3M additional trainable parameters in the IPS module.

Experimental results

Research questions

• RQ1 Can adaptive prompt generation improve SAM’s zero-shot performance in medical image segmentation without fine-tuning?
• RQ2 Can non-invasive pattern shifting in the mask decoder achieve state-of-the-art performance in MIS while preserving SAM’s prompting capability?
• RQ3 Is it possible to achieve effective domain adaptation in medical imaging using only a small number of learnable parameters, without fine-tuning the main model?
• RQ4 How does the combination of adaptive prompting and incremental pattern shifting compare to fine-tuning-based methods in terms of performance and stability?
• RQ5 Can the proposed method generalize across diverse medical imaging modalities and datasets?

Key findings

• ProMISe with both APM and IPS achieves state-of-the-art performance on Kvasir, EndoScene, ColonDB, ETIS, and ISIC2018 benchmarks using only 1.3M trainable parameters.
• Using SSFormer as a baseline, IPS improves mDice by up to 10% on ColonDB and 7% on ETIS with no fine-tuning.
• The Ablation Study confirms that the PaE module is essential for optimal IPS performance, with the full configuration (PaE + IPS tokens) achieving 0.911 mDice on Kvasir.
• Even with only IPS tokens (no PaE), performance improves significantly (0.862 mDice on Kvasir), showing the method’s robustness and flexibility.
• The framework maintains full interpretability through Euclidean prompts and avoids catastrophic forgetting, as all SAM parameters remain frozen.
• Incremental pattern shifting enables significant performance gains with increasing prompt points, suggesting potential for extension to coarse prompts like scribbles or sketches.