[Paper Review] Polyp-SAM++: Can A Text Guided SAM Perform Better for Polyp Segmentation?

Risab Biswas|arXiv (Cornell University)|Aug 12, 2023

Radiomics and Machine Learning in Medical Imaging9 citations

TL;DR

The paper investigates using a text-guided SAM (Polyp-SAM++) for polyp segmentation and compares its performance against unprompted SAM and various baselines on three colonoscopy datasets.

ABSTRACT

Meta recently released SAM (Segment Anything Model) which is a general-purpose segmentation model. SAM has shown promising results in a wide variety of segmentation tasks including medical image segmentation. In the field of medical image segmentation, polyp segmentation holds a position of high importance, thus creating a model which is robust and precise is quite challenging. Polyp segmentation is a fundamental task to ensure better diagnosis and cure of colorectal cancer. As such in this study, we will see how Polyp-SAM++, a text prompt-aided SAM, can better utilize a SAM using text prompting for robust and more precise polyp segmentation. We will evaluate the performance of a text-guided SAM on the polyp segmentation task on benchmark datasets. We will also compare the results of text-guided SAM vs unprompted SAM. With this study, we hope to advance the field of polyp segmentation and inspire more, intriguing research. The code and other details will be made publically available soon at https://github.com/RisabBiswas/Polyp-SAM++.

Motivation & Objective

• Assess whether text prompts can improve segmentation accuracy of SAM for colon polyp images.
• Quantitatively compare Polyp-SAM++ with unprompted SAM and state-of-the-art polyp segmentation models.
• Analyze qualitative results to understand robustness across diverse polyp appearances and imaging conditions.

Proposed method

• Use GroundingDINO to generate text-guided bounding boxes from a polyp-focused prompt.
• Feed the bounding box to SAM to obtain segmentation masks.
• Evaluate using mean dice (mDice), mean IoU (mIoU), and F-measure (Fm) across three datasets.
• Compare Polyp-SAM++ against CNN/ViT baselines and other SAM-based polyp methods.
• Analyze cases where Polyp-SAM++ fails and discuss potential improvements.

Figure 1 : Overview of the Polyp-SAM++ Architecture

Experimental results

Research questions

• RQ1 Does a text-guided prompting strategy improve SAM-based polyp segmentation compared to unprompted SAM?
• RQ2 How does Polyp-SAM++ perform relative to traditional polyp segmentation models on standard datasets?
• RQ3 What are the qualitative strengths and failure modes of text-guided SAM in polyp segmentation?

Key findings

• Polyp-SAM++ achieves competitive performance versus state-of-the-art polyp methods on three benchmark datasets.
• Text-guided localization helps SAM provide more accurate polyp segmentation by improving localization.
• Polyp-SAM++ outperforms unprompted SAM in several metrics across datasets, but still exhibits failures in challenging cases.
• Qualitative results show robustness of GroundingDINO + SAM in many scenarios, with identifiable failure examples discussed.

Figure 2 : Bounding Box created based on the Text-Prompt by GroundingDINO.