[Paper Review] Foundation Models for Biomedical Image Segmentation: A Survey

Ho Hin Lee, 裕二 池谷+10 more|arXiv (Cornell University)|Jan 15, 2024

Radiomics and Machine Learning in Medical Imaging13 citations

TL;DR

This survey analyzes the Segment Anything Model (SAM) and its medical image segmentation adaptations during the first six months after SAM’s publication, detailing zero-shot performance, domain-specific tuning, 3D extensions, and knowledge distillation across 33 open datasets.

ABSTRACT

Recent advancements in biomedical image analysis have been significantly driven by the Segment Anything Model (SAM). This transformative technology, originally developed for general-purpose computer vision, has found rapid application in medical image processing. Within the last year, marked by over 100 publications, SAM has demonstrated its prowess in zero-shot learning adaptations for medical imaging. The fundamental premise of SAM lies in its capability to segment or identify objects in images without prior knowledge of the object type or imaging modality. This approach aligns well with tasks achievable by the human visual system, though its application in non-biological vision contexts remains more theoretically challenging. A notable feature of SAM is its ability to adjust segmentation according to a specified resolution scale or area of interest, akin to semantic priming. This adaptability has spurred a wave of creativity and innovation in applying SAM to medical imaging. Our review focuses on the period from April 1, 2023, to September 30, 2023, a critical first six months post-initial publication. We examine the adaptations and integrations of SAM necessary to address longstanding clinical challenges, particularly in the context of 33 open datasets covered in our analysis. While SAM approaches or achieves state-of-the-art performance in numerous applications, it falls short in certain areas, such as segmentation of the carotid artery, adrenal glands, optic nerve, and mandible bone. Our survey delves into the innovative techniques where SAM's foundational approach excels and explores the core concepts in translating and applying these models effectively in diverse medical imaging scenarios.

Motivation & Objective

• Assess how SAM generalizes to medical imaging without domain-specific training.
• Survey domain-specific tuning techniques to adapt SAM for medical tasks.
• Examine 3D extensions and multi-dimensional adaptations of SAM for volumetric data.
• Explore knowledge distillation strategies leveraging SAM outputs for task-specific models.
• Identify clinical challenges and datasets where SAM succeeds or falls short.

Proposed method

• Review and categorize SAM adaptations into four main methodologies: zero-shot evaluation, adapters/projection/full tuning, 3D extension, and knowledge distillation.
• Describe SAM architecture: image encoder (MAE/ViT), prompt encoder, and light-weight mask decoder.
• Compile and analyze 33 open medical imaging datasets spanning multiple modalities and targets.
• Discuss 2D vs 3D handling, including 2.5D approaches and volumetric strategies for medical data.
• Summarize performance trends and limitations observed in early medical SAM studies.

Figure 1 : Evolution of SAM’s adaptation in medical research from April to September 2023. The graph showcases the cumulative studies emphasizing four phases: (i) Zero-shot Evaluation, (ii) Multi-dimensional Extension, (iii) Domain-specific Tuning, and (iv) Knowledge Distillation, highlighting a gro

Experimental results

Research questions

• RQ1 How well does SAM perform in zero-shot medical image segmentation across diverse datasets?
• RQ2 What domain-specific tuning strategies (projection, adapters, full tuning) best bridge SAM from natural images to medical imagery?
• RQ3 How are 3D medical imaging data addressed within SAM’s 2D framework, and what are the proposed extensions?
• RQ4 What role does knowledge distillation play in translating SAM capabilities to task-specific medical models?
• RQ5 What are the main anatomical targets and modalities where SAM excels or struggles, based on early public datasets?

Key findings

• SAM often achieves state-of-the-art or competitive results in various medical segmentation tasks, but has gaps in structures like carotid artery, adrenal glands, optic nerve, and mandible bone.
• Domain-specific tuning (adapter and full-tuning) and 3D extensions are active areas to improve SAM’s medical performance.
• 3D medical data are commonly processed as 2D slices or with 2.5D/volumetric approaches to accommodate SAM’s 2D design.
• Knowledge distillation using SAM outputs as pseudo labels can enhance downstream task models in semi-supervised settings.
• There is a broad dataset landscape with 33 public datasets across 7 modalities and 17 anatomies, illustrating SAM’s wide but uneven applicability.

(a) Data distribution by imaging modality