[Paper Review] One-shot Localization and Segmentation of Medical Images with Foundation Models

Deepa Anand, Gurunath Reddy M+11 more|arXiv (Cornell University)|Oct 28, 2023

Radiomics and Machine Learning in Medical Imaging7 citations

TL;DR

The paper shows that pre-trained natural-image foundation models (ViT variants and Stable Diffusion) can enable single-shot localization and segmentation on diverse medical images by guiding a Segment Anything model with a template image, achieving competitive Dice scores across modalities and outperforming a recent few-shot method in most tasks.

ABSTRACT

Recent advances in Vision Transformers (ViT) and Stable Diffusion (SD) models with their ability to capture rich semantic features of the image have been used for image correspondence tasks on natural images. In this paper, we examine the ability of a variety of pre-trained ViT (DINO, DINOv2, SAM, CLIP) and SD models, trained exclusively on natural images, for solving the correspondence problems on medical images. While many works have made a case for in-domain training, we show that the models trained on natural images can offer good performance on medical images across different modalities (CT,MR,Ultrasound) sourced from various manufacturers, over multiple anatomical regions (brain, thorax, abdomen, extremities), and on wide variety of tasks. Further, we leverage the correspondence with respect to a template image to prompt a Segment Anything (SAM) model to arrive at single shot segmentation, achieving dice range of 62%-90% across tasks, using just one image as reference. We also show that our single-shot method outperforms the recently proposed few-shot segmentation method - UniverSeg (Dice range 47%-80%) on most of the semantic segmentation tasks(six out of seven) across medical imaging modalities.

Motivation & Objective

• Assess whether pre-trained natural-image models can solve medical image correspondence tasks without in-domain training.
• Demonstrate a single-shot segmentation pipeline using a template image to prompt a SAM model.
• Evaluate performance across multiple modalities (CT, MR, Ultrasound) and anatomical regions.
• Compare single-shot performance to recent few-shot segmentation methods in medical imaging.

Proposed method

• Evaluate a variety of pre-trained ViT models (DINO, DINOv2, SAM, CLIP) and Stable Diffusion models trained on natural images for medical image correspondence tasks.
• Use template-image correspondence to prompt a Segment Anything model to obtain single-shot segmentation.
• Report Dice scores across multiple modalities and anatomical regions to assess generalization.
• Compare the proposed single-shot method against UniverSeg (few-shot) on semantic segmentation tasks.
• Present results demonstrating cross-modality and cross-manufacturer robustness.

Experimental results

Research questions

• RQ1 Can foundation models trained on natural images perform medical image correspondence tasks without domain-specific fine-tuning?
• RQ2 Is a template-image guided prompting approach able to yield accurate one-shot segmentation in medical images?
• RQ3 How does single-shot segmentation performance compare to recent few-shot methods across modalities and tasks?

Key findings

• Single-shot localization and segmentation using natural-image foundation models with a template prompt yields Dice scores in the range of 62%–90% across tasks.
• The method uses just one reference image to prompt SAM for segmentation.
• On most semantic segmentation tasks across modalities (six of seven), the single-shot method outperforms the few-shot UniverSeg approach (Dice range 47%–80%).
• The experiments cover CT, MR, and Ultrasound across brain, thorax, abdomen, and extremities from various manufacturers.