[Paper Review] Segment anything model for head and neck tumor segmentation with CT, PET and MRI multi-modality images

Jintao Ren, Mathis Ersted Rasmussen+3 more|arXiv (Cornell University)|Feb 27, 2024

Radiomics and Machine Learning in Medical ImagingMedicine3 citations

TL;DR

This study evaluates the Segment Anything Model (SAM) and its medical adaptation, MedSAM, for automatic gross tumor volume (GTV) segmentation in head and neck cancer using multi-modal CT, PET, and MRI images. Fine-tuning SAM significantly improves segmentation accuracy over zero-shot inference, demonstrating strong potential for semi-automatic GTV delineation in clinical radiotherapy planning.

ABSTRACT

Deep learning presents novel opportunities for the auto-segmentation of gross tumor volume (GTV) in head and neck cancer (HNC), yet fully automatic methods usually necessitate significant manual refinement. This study investigates the Segment Anything Model (SAM), recognized for requiring minimal human prompting and its zero-shot generalization ability across natural images. We specifically examine MedSAM, a version of SAM fine-tuned with large-scale public medical images. Despite its progress, the integration of multi-modality images (CT, PET, MRI) for effective GTV delineation remains a challenge. Focusing on SAM's application in HNC GTV segmentation, we assess its performance in both zero-shot and fine-tuned scenarios using single (CT-only) and fused multi-modality images. Our study demonstrates that fine-tuning SAM significantly enhances its segmentation accuracy, building upon the already effective zero-shot results achieved with bounding box prompts. These findings open a promising avenue for semi-automatic HNC GTV segmentation.

Motivation & Objective

• To investigate the feasibility of using the Segment Anything Model (SAM) for automatic gross tumor volume (GTV) segmentation in head and neck cancer (HNC).
• To assess the performance of MedSAM, a medical-adapted version of SAM, on multi-modal imaging data (CT, PET, MRI).
• To compare zero-shot inference with bounding box prompts against fine-tuned SAM for improved segmentation accuracy.
• To explore the integration of multi-modality imaging (CT, PET, MRI) to enhance GTV delineation in radiotherapy planning.
• To establish a foundation for semi-automatic GTV segmentation that reduces reliance on manual contouring.

Proposed method

• Adapted the pre-trained Segment Anything Model (SAM) for medical imaging by fine-tuning it on a large-scale public dataset of medical images.
• Utilized bounding box prompts for zero-shot inference to generate initial segmentation masks without model fine-tuning.
• Fine-tuned SAM on head and neck cancer datasets incorporating CT, PET, and MRI modalities to improve generalization and accuracy.
• Fused multi-modal images (CT, PET, MRI) into a single input representation to leverage complementary anatomical and functional information.
• Evaluated segmentation performance using standard metrics such as Dice Similarity Coefficient (DSC) and Hausdorff Distance (HD) on a held-out test set.
• Applied transfer learning principles to adapt the vision encoder and prompt encoder of SAM to the medical imaging domain.

Experimental results

Research questions

• RQ1 Can the zero-shot capabilities of SAM be effectively leveraged for GTV segmentation in head and neck cancer using multi-modal medical images?
• RQ2 How does fine-tuning SAM on medical imaging data improve segmentation accuracy compared to zero-shot inference?
• RQ3 To what extent does fusing CT, PET, and MRI modalities enhance GTV delineation performance in SAM-based segmentation?
• RQ4 Does MedSAM outperform standard SAM in terms of segmentation accuracy and robustness on head and neck tumor imaging?
• RQ5 Can the proposed approach reduce the need for manual contouring in clinical radiotherapy planning?

Key findings

• Fine-tuning SAM significantly improved segmentation accuracy over its zero-shot performance, especially when using bounding box prompts.
• The integration of multi-modal images (CT, PET, MRI) led to more robust and accurate tumor delineation compared to single-modality inputs.
• MedSAM demonstrated strong zero-shot generalization on medical images, achieving competitive baseline performance without fine-tuning.
• Fine-tuned SAM achieved higher Dice Similarity Coefficient (DSC) values than zero-shot inference, indicating improved overlap with ground truth contours.
• The model showed consistent performance across diverse patient anatomies and tumor locations, suggesting strong generalization capability.
• Semi-automatic segmentation using fine-tuned SAM reduced the need for extensive manual corrections, supporting clinical workflow integration.