[论文解读] Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions

Yichi Zhang, Zhenrong Shen+1 more|arXiv (Cornell University)|Jan 7, 2024

Artificial Intelligence in Healthcare and Education被引用 8

一句话总结

tldr: 对 Segment Anything Model (SAM) 在零-shot 设置下对医学影像分割的表现的调查，以及研究者如何通过微调、自动提示和3D扩展等方法将 SAM 应用于医学数据。

ABSTRACT

Due to the inherent flexibility of prompting, foundation models have emerged as the predominant force in the fields of natural language processing and computer vision. The recent introduction of the Segment Anything Model (SAM) signifies a noteworthy expansion of the prompt-driven paradigm into the domain of image segmentation, thereby introducing a plethora of previously unexplored capabilities. However, the viability of its application to medical image segmentation remains uncertain, given the substantial distinctions between natural and medical images. In this work, we provide a comprehensive overview of recent endeavors aimed at extending the efficacy of SAM to medical image segmentation tasks, encompassing both empirical benchmarking and methodological adaptations. Additionally, we explore potential avenues for future research directions in SAM's role within medical image segmentation. While direct application of SAM to medical image segmentation does not yield satisfactory performance on multi-modal and multi-target medical datasets so far, numerous insights gleaned from these efforts serve as valuable guidance for shaping the trajectory of foundational models in the realm of medical image analysis. To support ongoing research endeavors, we maintain an active repository that contains an up-to-date paper list and a succinct summary of open-source projects at https://github.com/YichiZhang98/SAM4MIS.

研究动机与目标

• 在医学领域介绍基础模型与 SAM 架构用于图像分割。
• 评估 SAM 在各种医学影像模态与任务中的零-shot 性能。
• 总结包括微调、提示工程、自动提示和框架修改在内的适配策略。
• 突出医学影像分析中 SAM 的挑战、局限性及潜在未来方向。

提出的方法

• 描述 SAM 架构：图像编码器（ViT）、提示编码器，以及轻量级掩码解码器。
• 总结对 CT、MRI、病理、ец://请忽略无关文本/

实验结果

研究问题

• RQ1 How well does SAM perform in zero-shot medical image segmentation across different modalities and tasks?
• RQ2 What adaptation strategies (fine-tuning, PEFT, auto-prompting, 3D extensions) improve SAM’s segmentation performance on medical data?
• RQ3 What are the limitations of SAM when applied directly to medical images, and which factors (dimensionality, modality, size, contrast) influence performance?
• RQ4 Can auto-prompting and uncertainty estimation yield reliable, annotation-efficient medical segmentation with SAM?
• RQ5 What future directions and open problems exist for integrating SAM into clinical radiotherapy, surgery, and multi-modal analysis?

主要发现

• SAM’s zero-shot medical segmentation generally underperforms on many tasks without prompts or fine-tuning.
• Fine-tuning (including full and parameter-efficient) and specialized medical datasets (e.g., MedSAM) substantially improve performance, sometimes surpassing or matching SOTA models on selected tasks.
• Auto-prompting and learnable prompts (e.g., AutoSAM, DeSAM, UR-SAM) can enhance robustness and enable fully automatic or semi-automatic segmentation with improved reliability.
• 3D adaptations (e.g., SAM-Med3D, 3DSAM-adapter) address the limitations of 2D SAM for volumetric data, improving performance on 3D medical images.
• Uncertainty-guided prompt generation and rectification (UR-SAM, EviPrompt) improve reliability across prompts and domains.
• Hybrid frameworks integrating SAM with task-specific models (nnSAM, SAMUS, SAMPath) show improved segmentation accuracy and practicality in clinical workflows.