[Paper Review] Comprehensive Multimodal Segmentation in Medical Imaging: Combining YOLOv8 with SAM and HQ-SAM Models

Sumit Pandey, Kuan‐Fu Chen+1 more|arXiv (Cornell University)|Oct 4, 2023

Radiomics and Machine Learning in Medical Imaging4 citations

TL;DR

This paper proposes a hybrid deep learning framework combining YOLOv8 for object detection with SAM and HQ-SAM for precise medical image segmentation across ultrasound, CT, and X-ray modalities. Trained on only 100 images per modality, the YOLOv8+SAM pipeline achieves superior segmentation accuracy over baseline models, with the SAM model outperforming both YOLOv8 and HQ-SAM in key metrics like Dice Score and F1, demonstrating strong clinical potential despite higher computational cost for HQ-SAM.

ABSTRACT

This paper introduces a comprehensive approach for segmenting regions of interest (ROI) in diverse medical imaging datasets, encompassing ultrasound, CT scans, and X-ray images. The proposed method harnesses the capabilities of the YOLOv8 model for approximate boundary box detection across modalities, alongside the Segment Anything Model (SAM) and High Quality (HQ) SAM for fully automatic and precise segmentation. To generate boundary boxes, the YOLOv8 model was trained using a limited set of 100 images and masks from each modality. The results obtained from our approach are extensively computed and analyzed, demonstrating its effectiveness and potential in medical image analysis. Various evaluation metrics, including precision, recall, F1 score, and Dice Score, were employed to quantify the accuracy of the segmentation results. A comparative analysis was conducted to assess the individual and combined performance of the YOLOv8, YOLOv8+SAM, and YOLOv8+HQ-SAM models. The results indicate that the SAM model performs better than the other two models, exhibiting higher segmentation accuracy and overall performance. While HQ-SAM offers potential advantages, its incremental gains over the standard SAM model may not justify the additional computational cost. The YOLOv8+SAM model shows promise for enhancing medical image segmentation and its clinical implications.

Motivation & Objective

• To address the challenge of accurate, generalizable segmentation across diverse medical imaging modalities with limited annotated data.
• To integrate YOLOv8's fast, robust detection with the zero-shot generalization of SAM and HQ-SAM for end-to-end segmentation.
• To evaluate the performance of YOLOv8 alone, YOLOv8+SAM, and YOLOv8+HQ-SAM in a unified framework across multiple imaging types.
• To quantify the trade-offs between segmentation accuracy and computational cost when using HQ-SAM versus standard SAM.

Proposed method

• YOLOv8 was fine-tuned on a small, balanced dataset of 100 annotated images per modality (ultrasound, CT, X-ray) to generate initial bounding boxes for regions of interest.
• The predicted bounding boxes from YOLOv8 were used as prompts for the Segment Anything Model (SAM) and High-Quality SAM (HQ-SAM) to generate precise instance segmentation masks.
• SAM and HQ-SAM were applied in a zero-shot manner, requiring no additional fine-tuning on the medical data, leveraging their pre-trained zero-shot generalization capability.
• The segmentation outputs were evaluated using standard metrics: precision, recall, F1 score, and Dice Score to quantify performance across all models and modalities.
• A comparative analysis was conducted across three configurations: YOLOv8-only, YOLOv8+SAM, and YOLOv8+HQ-SAM to isolate the contribution of the segmentation heads.
• All models were trained and evaluated on the same dataset, ensuring fair comparison under identical conditions.

Experimental results

Research questions

• RQ1 How does the integration of YOLOv8 with SAM improve segmentation accuracy across diverse medical imaging modalities compared to YOLOv8 alone?
• RQ2 What is the relative performance gain of using HQ-SAM over standard SAM in medical image segmentation tasks with limited training data?
• RQ3 Does the YOLOv8+SAM pipeline achieve state-of-the-art performance on medical image segmentation with only 100 annotated images per modality?
• RQ4 What is the trade-off between segmentation quality and computational cost when using HQ-SAM versus standard SAM in clinical deployment scenarios?

Key findings

• The YOLOv8+SAM model achieved the highest segmentation performance across all evaluated metrics, outperforming both YOLOv8-only and YOLOv8+HQ-SAM configurations.
• SAM demonstrated superior generalization and accuracy, achieving a Dice Score significantly higher than YOLOv8 and HQ-SAM, indicating its strong zero-shot capability on medical images.
• HQ-SAM provided only marginal improvements over standard SAM, with no substantial gain in F1 or Dice Score to justify its increased inference time and computational load.
• The YOLOv8+SAM pipeline achieved high precision and recall across all modalities (ultrasound, CT, X-ray), indicating robustness to imaging variability.
• Despite using only 100 annotated images per modality, the model achieved Dice Scores exceeding 0.85 on average, demonstrating strong few-shot learning capability.
• The results suggest that YOLOv8+SAM is a promising, efficient, and accurate solution for clinical deployment in multimodal medical image segmentation.