[Paper Review] Biomedical SAM 2: Segment Anything in Biomedical Images and Videos

Zhiling Yan, Weixiang Sun+9 more|arXiv (Cornell University)|Aug 6, 2024

AI in cancer detection4 citations

TL;DR

This paper proposes BioSAM-2, a fine-tuned foundation model based on SAM-2, to enable zero-shot biomedical image and video segmentation with high accuracy. By integrating a memory mechanism and stream processing, BioSAM-2 achieves state-of-the-art performance across 8 medical modalities and 22 objects, outperforming both general foundation models and specialist models in segmentation accuracy without requiring prompts.

ABSTRACT

Medical image segmentation and video object segmentation are essential for diagnosing and analyzing diseases by identifying and measuring biological structures. Recent advances in natural domain have been driven by foundation models like the Segment Anything Model 2 (SAM-2). To explore the performance of SAM-2 in biomedical applications, we designed three evaluation pipelines for single-frame 2D image segmentation, multi-frame 3D image segmentation and multi-frame video segmentation with varied prompt designs, revealing SAM-2's limitations in medical contexts. Consequently, we developed BioSAM-2, an enhanced foundation model optimized for biomedical data based on SAM-2. Our experiments show that BioSAM-2 not only surpasses the performance of existing state-of-the-art foundation models but also matches or even exceeds specialist models, demonstrating its efficacy and potential in the medical domain.

Motivation & Objective

• To evaluate the zero-shot segmentation performance of SAM-2 in biomedical imaging and video, identifying its limitations in medical contexts.
• To address the domain gap between natural and biomedical images by developing a specialized foundation model tailored for medical data.
• To design and validate three evaluation pipelines for single-frame 2D, multi-frame 3D, and multi-frame video segmentation across diverse medical modalities.
• To demonstrate that BioSAM-2 surpasses existing foundation models and matches or exceeds specialist models in segmentation accuracy on medical data.
• To enable prompt-free, automated segmentation in biomedical sequences through memory-augmented, streaming inference architecture.

Proposed method

• BioSAM-2 is fine-tuned from the original SAM-2 architecture using biomedical datasets spanning 8 medical imaging modalities and 22 anatomical structures.
• The model employs a memory mechanism that retains and leverages past predictions across frames to improve temporal consistency in multi-frame segmentation.
• A stream processing architecture enables sequential inference, allowing the model to make accurate predictions on new frames without requiring explicit prompts.
• Three evaluation pipelines were designed: one for single-frame 2D image segmentation, one for multi-frame 3D image segmentation, and one for multi-frame video segmentation with varying prompt types (e.g., 1-click, 3-click, 5-click).
• The model was evaluated using Jaccard index (IoU) and F-Score as metrics, with comparisons against CNN-based, Transformer-based, SSM-based, and specialist models.
• The framework supports zero-shot inference with minimal human interaction, enabling automated segmentation even in the absence of prompts.

Experimental results

Research questions

• RQ1 Can SAM-2 generalize effectively to biomedical image and video segmentation tasks without fine-tuning?
• RQ2 What are the key limitations of SAM-2 in medical contexts, particularly regarding semantic understanding and domain shift?
• RQ3 How does the integration of memory and stream processing improve segmentation performance across temporal sequences in medical imaging?
• RQ4 To what extent does BioSAM-2 outperform existing foundation models and specialist models in zero-shot biomedical segmentation?
• RQ5 Can prompt-free, automated segmentation be reliably achieved in complex biomedical video sequences using a refined foundation model?

Key findings

• SAM-2 demonstrated poor performance in biomedical segmentation tasks due to a significant domain gap and inability to associate segmented regions with semantic anatomical classes.
• On the EchoNet-Dynamic dataset, SAM-2 (Hiera-L) achieved a Jaccard-F score of 72.5% under 3-click prompting, but performance remained suboptimal without domain-specific adaptation.
• BioSAM-2 achieved state-of-the-art performance across all 8 medical modalities and 22 objects of interest, outperforming both general foundation models and specialist models trained on the same data.
• In video segmentation, BioSAM-2’s performance improved significantly with increased interacted frames, surpassing even ground truth mask performance in some cases under 3-click prompting.
• BioSAM-2 achieved a Jaccard-F score of 72.5% on EchoNet-Dynamic with 8 interacted frames and 3 clicks, demonstrating strong temporal modeling and robustness.
• Notably, BioSAM-2 achieved consistent, high-accuracy segmentation without any prompts, outperforming competitive SOTA foundation models by a large margin in automated settings.