[Paper Review] Segment Anything in Medical Images

Jun Ma, Yuting He+4 more|arXiv (Cornell University)|Apr 24, 2023

Radiomics and Machine Learning in Medical ImagingMedicine41 references87 citations

TL;DR

MedSAM is the first foundation model for universal medical image segmentation, trained on over one million image-mask pairs, and it outperforms prior segmentation foundations and rivals specialist models across many tasks while enabling precise tumor burden quantification.

ABSTRACT

Medical image segmentation is a critical component in clinical practice, facilitating accurate diagnosis, treatment planning, and disease monitoring. However, existing methods, often tailored to specific modalities or disease types, lack generalizability across the diverse spectrum of medical image segmentation tasks. Here we present MedSAM, a foundation model designed for bridging this gap by enabling universal medical image segmentation. The model is developed on a large-scale medical image dataset with 1,570,263 image-mask pairs, covering 10 imaging modalities and over 30 cancer types. We conduct a comprehensive evaluation on 86 internal validation tasks and 60 external validation tasks, demonstrating better accuracy and robustness than modality-wise specialist models. By delivering accurate and efficient segmentation across a wide spectrum of tasks, MedSAM holds significant potential to expedite the evolution of diagnostic tools and the personalization of treatment plans.

Motivation & Objective

• Motivate the need for a universal medical image segmentation model with broad generalization across modalities and tasks.
• Develop MedSAM by adapting the Segment Anything Model (SAM) to a large-scale medical dataset.
• Evaluate MedSAM against state-of-the-art segmentation foundations and specialist models on internal and external tasks.
• Demonstrate MedSAM's ability to quantify tumor burden accurately and compare with expert performance.

Proposed method

• Adapt SAM architecture with image encoder, prompt encoder, and mask decoder using a ViT-based image encoder.
• Train on 1,090,486 medical image-mask pairs across 15 modalities and 30+ cancer types.
• Use bounding box prompts; simulate prompts from ground-truth masks with perturbations for training.
• Pre-train with masked auto-encoder then fully supervise on the SAM dataset; fine-tune image encoder and mask decoder while fixing the prompt encoder.
• Optimize with AdamW using Dice and cross-entropy losses combined as the final loss; train on 20 A100 GPUs for 100 epochs.

Experimental results

Research questions

• RQ1 Can a single foundation model be trained to generalize across diverse medical imaging modalities and segmentation tasks?
• RQ2 How does MedSAM compare to SAM and specialist U-Net models on internal validation tasks?
• RQ3 How well does MedSAM generalize to external validation datasets and unseen tasks/modalities?
• RQ4 Can MedSAM enable accurate tumor burden quantification across cancers?
• RQ5 How does MedSAM's segmentation performance compare to human experts in specific tasks?

Key findings

• MedSAM outperforms the state-of-the-art segmentation foundation model and rivals or surpasses specialist models on many tasks.
• On internal validation, MedSAM achieves high median Dice scores across key tasks (e.g., intracranial hemorrhage CT, glioma MR T1, pneumothorax CXR, polyp endoscopy) and outperforms SAM and many U-Net baselines.
• External validation shows MedSAM maintaining superior performance on unseen datasets and modalities, with notable improvements over SAM and specialist models.
• MedSAM enables precise tumor burden quantification with a Pearson correlation of r=0.99 between MedSAM-derived and expert volumes for selected cancers.
• In prostate MR segmentation, MedSAM matches or surpasses six human experts, illustrating clinical competitiveness.