[Paper Review] SAM3D: Segment Anything Model in Volumetric Medical Images

Nhat-Tan Bui, Dinh-Hieu Hoang+6 more|arXiv (Cornell University)|Sep 7, 2023

Radiomics and Machine Learning in Medical ImagingMedicine3 citations

TL;DR

SAM3D introduces a lightweight, efficient 3D medical image segmentation model that leverages the pre-trained SAM encoder for volumetric data by processing 2D slices first and then applying a 3D decoder to capture depth-wise relationships. It achieves state-of-the-art performance with only 1.88M parameters, outperforming other SAM-based and SOTA models in segmentation accuracy while significantly reducing model complexity.

ABSTRACT

Image segmentation remains a pivotal component in medical image analysis, aiding in the extraction of critical information for precise diagnostic practices. With the advent of deep learning, automated image segmentation methods have risen to prominence, showcasing exceptional proficiency in processing medical imagery. Motivated by the Segment Anything Model (SAM)-a foundational model renowned for its remarkable precision and robust generalization capabilities in segmenting 2D natural images-we introduce SAM3D, an innovative adaptation tailored for 3D volumetric medical image analysis. Unlike current SAM-based methods that segment volumetric data by converting the volume into separate 2D slices for individual analysis, our SAM3D model processes the entire 3D volume image in a unified approach. Extensive experiments are conducted on multiple medical image datasets to demonstrate that our network attains competitive results compared with other state-of-the-art methods in 3D medical segmentation tasks while being significantly efficient in terms of parameters. Code and checkpoints are available at https://github.com/UARK-AICV/SAM3D.

Motivation & Objective

• To address the limitations of existing 3D medical segmentation models that rely on complex architectures and high parameter counts.
• To extend the capabilities of the Segment Anything Model (SAM) to volumetric medical images without fine-tuning the entire model.
• To improve segmentation performance by capturing 3D spatial relationships across slices, rather than processing slices in isolation.
• To develop a computationally efficient and parameter-light framework that maintains high accuracy in 3D medical image segmentation.

Proposed method

• Freeze the pre-trained SAM image encoder (ViT-B) to retain robust low-level features such as edges and boundaries from natural images.
• Process each 2D slice of a 3D volume independently through the frozen SAM encoder to generate 3D slice embeddings of shape $ rac{H}{16} imes rac{W}{16} imes D imes 256 $.
• Apply a lightweight 3D convolutional decoder to model depth-wise relationships between slice embeddings, enabling 3D context understanding.
• Remove the prompt encoder from SAM to avoid modality-specific biases and ensure generalization across diverse medical imaging modalities.
• Train only the 3D decoder head end-to-end while keeping the SAM encoder frozen, minimizing parameter updates and computational cost.
• Incorporate skip connections in the 3D decoder to preserve fine-grained features and improve edge and boundary localization in segmentation.

Experimental results

Research questions

• RQ1 Can a pre-trained 2D foundation model like SAM be effectively adapted for 3D volumetric medical image segmentation without full fine-tuning?
• RQ2 Does capturing inter-slice relationships via a 3D decoder improve segmentation performance compared to slice-by-slice processing?
• RQ3 Can a lightweight 3D decoder achieve competitive results with significantly fewer parameters than existing SOTA 3D segmentation models?
• RQ4 How does SAM3D compare in performance and efficiency to other SAM-based and Transformer/CNN hybrid models on multi-modality 3D medical datasets?

Key findings

• SAM3D achieves a DSC of 77.78% on the Synapse dataset, outperforming SAMed_s (6.32M params) by 1.78% despite using only 1.88M parameters.
• On the ACDC cardiac dataset, SAM3D improves DSC by 0.41% over TransUNet while using less than 1/50th the number of parameters.
• On the BraTS brain tumor dataset, SAM3D achieves a 1.8% higher DSC than UNETR, with less than 1/20th the parameter count.
• The ablation study confirms that skip connections in the 3D decoder improve performance, highlighting their role in preserving boundary details.
• SAM3D demonstrates strong generalization across diverse modalities and anatomical structures, including cardiac, brain, and lung segmentation tasks.
• The model’s performance remains competitive even when using the smallest SAM variant (ViT-B), suggesting potential for further gains with larger backbone variants (e.g., ViT-L, ViT-H).