[Paper Review] Can SAM Segment Polyps?

Tao Zhou, Yizhe Zhang+3 more|arXiv (Cornell University)|Apr 15, 2023

Radiomics and Machine Learning in Medical ImagingMedicine9 citations

TL;DR

The paper evaluates Segment Anything Model (SAM) in unprompted polyp segmentation on five colonoscopy datasets and finds SAM underperforms compared with state-of-the-art polyp segmentation models; fine-tuning SAM on task-specific data may improve results.

ABSTRACT

Recently, Meta AI Research releases a general Segment Anything Model (SAM), which has demonstrated promising performance in several segmentation tasks. As we know, polyp segmentation is a fundamental task in the medical imaging field, which plays a critical role in the diagnosis and cure of colorectal cancer. In particular, applying SAM to the polyp segmentation task is interesting. In this report, we evaluate the performance of SAM in segmenting polyps, in which SAM is under unprompted settings. We hope this report will provide insights to advance this polyp segmentation field and promote more interesting works in the future. This project is publicly at https://github.com/taozh2017/SAMPolyp.

Motivation & Objective

• Assess how well SAM, in unprompted mode, segments polyps in colonoscopy images without user prompts.
• Compare SAM to state-of-the-art CNN/Transformer-based polyp segmentation models on multiple benchmarks.
• Analyze qualitative segmentation examples to identify strengths and failure modes of SAM in medical imaging.
• Provide guidance for future work on applying SAM to polyp segmentation and potential fine-tuning strategies.

Proposed method

• Use SAM with two backbones (SAM-H and SAM-L) to segment polyps in five datasets.
• Evaluate segmentation performance with six metrics: mDice, mIoU, S_alpha, F_beta^w, E_phi^max, and M.
• Select the best mask from SAM’s multiple outputs per image by maximizing S_alpha against ground-truth.
• Compare SAM against 14 CNN-based models and two Transformer-based models on each dataset.
• Provide qualitative visualization of successful and failed SAM segmentations.
• Discuss a potential fine-tuning approach to improve polyp segmentation performance.

Experimental results

Research questions

• RQ1 Can SAM, in unprompted mode, achieve competitive polyp segmentation performance on standard colonoscopy benchmarks?
• RQ2 How does SAM compare to established CNN/Transformer-based polyp segmentation methods across diverse datasets?
• RQ3 What are the common failure modes of SAM when segmenting polyps in colonoscopy images?
• RQ4 Would fine-tuning SAM on polyp-specific datasets improve segmentation performance?

Key findings

• SAM under unprompted settings does not outperform dedicated polyp segmentation models across the tested datasets.
• On datasets like CVC-ClinicDB, Kvasir, CVC-ColonDB, ETIS, and CVC-300, SAM variants show lower performance across multiple metrics compared to state-of-the-art methods.
• Qualitative results show SAM can segment some polyps well, but struggles when polyp boundaries are blurred or non-sharp relative to surrounding mucosa.
• Failure cases illustrate SAM's limitations in medical image segmentation when trained on natural-image corpora.
• The authors suggest fine-tuning SAM on task-specific polyp datasets could yield better segmentation performance than zero-shot use.