[Paper Review] SamDSK: Combining Segment Anything Model with Domain-Specific Knowledge for Semi-Supervised Learning in Medical Image Segmentation

Yizhe Zhang, Tao Zhou+4 more|arXiv (Cornell University)|Aug 26, 2023

AI in cancer detection6 citations

TL;DR

SamDSK proposes an iterative semi-supervised learning framework that combines the Segment Anything Model (SAM) with domain-specific knowledge (DSK) to improve medical image segmentation using minimal labeled data. By using optimal matching between SAM-generated proposals and pixel-level/image-level DSK, it progressively expands the labeled set, achieving state-of-the-art performance on breast cancer, polyp, and skin lesion segmentation with only 10% labeled data.

ABSTRACT

The Segment Anything Model (SAM) exhibits a capability to segment a wide array of objects in natural images, serving as a versatile perceptual tool for various downstream image segmentation tasks. In contrast, medical image segmentation tasks often rely on domain-specific knowledge (DSK). In this paper, we propose a novel method that combines the segmentation foundation model (i.e., SAM) with domain-specific knowledge for reliable utilization of unlabeled images in building a medical image segmentation model. Our new method is iterative and consists of two main stages: (1) segmentation model training; (2) expanding the labeled set by using the trained segmentation model, an unlabeled set, SAM, and domain-specific knowledge. These two stages are repeated until no more samples are added to the labeled set. A novel optimal-matching-based method is developed for combining the SAM-generated segmentation proposals and pixel-level and image-level DSK for constructing annotations of unlabeled images in the iterative stage (2). In experiments, we demonstrate the effectiveness of our proposed method for breast cancer segmentation in ultrasound images, polyp segmentation in endoscopic images, and skin lesion segmentation in dermoscopic images. Our work initiates a new direction of semi-supervised learning for medical image segmentation: the segmentation foundation model can be harnessed as a valuable tool for label-efficient segmentation learning in medical image segmentation.

Motivation & Objective

• To address the challenge of limited annotated medical images by leveraging abundant unlabeled data through semi-supervised learning.
• To integrate domain-specific knowledge (DSK) with the Segment Anything Model (SAM) for reliable pseudo-labeling of unlabeled medical images.
• To develop an iterative framework that jointly trains a segmentation model and expands the labeled set using SAM and DSK.
• To improve segmentation performance in low-shot settings by combining foundation model predictions with task-specific constraints.
• To establish a robust, label-efficient method for medical image segmentation applicable across diverse imaging modalities.

Proposed method

• The method uses an iterative two-stage process: (1) training a segmentation model on an initial labeled set, and (2) expanding the labeled set using unlabeled images.
• In stage (2), the trained model generates pixel-level DSK (segmentation probability maps), while image-level DSK enforces constraints like the number of expected regions of interest per image.
• SAM generates segmentation proposals for each unlabeled image, which are then optimally matched with the DSK using a novel matching-based annotation construction method.
• The optimal matching process selects high-quality proposals that best align with both the model’s predictions and domain-specific constraints.
• The expanded labeled set is used to retrain the segmentation model, and the process repeats until no further samples are added.
• The framework integrates SAM’s zero-shot generalization with medical domain priors to improve pseudo-label quality and model robustness.

Figure 1: A high-level comparison between a conventional approach (left) and our proposed SamDSK (right) for semi-supervised learning (SSL) in medical image segmentation.

Experimental results

Research questions

• RQ1 Can SAM be effectively combined with domain-specific knowledge to improve pseudo-labeling quality in medical image segmentation?
• RQ2 How does iterative model retraining with DSK-augmented pseudo-labels affect segmentation performance in low-data regimes?
• RQ3 Does optimal matching between SAM proposals and DSK outperform uncertainty-based or confidence-based selection for sample expansion?
• RQ4 How does the performance of SAM vary across different medical imaging modalities, such as ultrasound, endoscopy, and dermoscopy?
• RQ5 Can the integration of image-level and pixel-level DSK lead to more consistent and accurate segmentation annotations than using SAM alone?

Key findings

• SamDSK achieved state-of-the-art performance on breast cancer segmentation in ultrasound images using only 10% of the labeled data, with significant mIoU improvements over baseline and competing SSL methods.
• For polyp segmentation in endoscopic images, SamDSK achieved a 95% mIoU on the top 10% of samples selected via optimal matching, demonstrating high-quality pseudo-label selection.
• In skin lesion segmentation on the ISIC 2018 dataset, SamDSK outperformed competing methods, achieving a 92% mIoU on the highest-quality selected samples using only 10% labeled data.
• The optimal matching-based selection using the β score outperformed model uncertainty and confidence in identifying high-quality samples, with statistically significant gains in segmentation quality.
• SAM-generated proposals achieved a Dice coefficient of 89.1% when matched to ground truth in polyp segmentation, but only 64.4% in malignant breast cancer cases, indicating modality-dependent performance.
• The multi-round iterative process consistently improved model performance across all datasets, confirming the stability and scalability of the SamDSK framework.

Figure 2: The main steps of our proposed SamDSK method. Step-0 to step-4 are repeated iteratively until a certain criterion is met (e.g., no more samples are added to the labeled set). Step-0 may need to run only once if SAM remains unchanged during the process.