[Paper Review] SegMatch: A semi-supervised learning method for surgical instrument segmentation

Wei Meng, Charlie Budd+4 more|arXiv (Cornell University)|Aug 9, 2023

Medical Imaging and Analysis4 citations

TL;DR

SegMatch proposes a semi-supervised learning framework for surgical instrument segmentation that adapts FixMatch for dense prediction tasks by incorporating equivariance-invariant augmentations and a learnable adversarial augmentation strategy via I-FGSM. It achieves state-of-the-art performance on Robust-MIS 2019 and EndoVis 2017, surpassing fully supervised methods using only 30% labeled data and 17K unannotated images.

ABSTRACT

Surgical instrument segmentation is recognised as a key enabler in providing advanced surgical assistance and improving computer-assisted interventions. In this work, we propose SegMatch, a semi-supervised learning method to reduce the need for expensive annotation for laparoscopic and robotic surgical images. SegMatch builds on FixMatch, a widespread semi supervised classification pipeline combining consistency regularization and pseudo-labelling, and adapts it for the purpose of segmentation. In our proposed SegMatch, the unlabelled images are first weakly augmented and fed to the segmentation model to generate pseudo-labels. In parallel, images are fed to a strong augmentation branch and consistency between the branches is used as an unsupervised loss. To increase the relevance of our strong augmentations, we depart from using only handcrafted augmentations and introduce a trainable adversarial augmentation strategy. Our FixMatch adaptation for segmentation tasks further includes carefully considering the equivariance and invariance properties of the augmentation functions we rely on. For binary segmentation tasks, our algorithm was evaluated on the MICCAI Instrument Segmentation Challenge datasets, Robust-MIS 2019 and EndoVis 2017. For multi-class segmentation tasks, we relied on the recent CholecInstanceSeg dataset. Our results show that SegMatch outperforms fully-supervised approaches by incorporating unlabelled data, and surpasses a range of state-of-the-art semi-supervised models across different labelled to unlabelled data ratios.

Motivation & Objective

• To reduce reliance on expensive manual annotations in surgical instrument segmentation by leveraging unlabelled laparoscopic and robotic surgical images.
• To adapt the semi-supervised classification framework FixMatch for semantic segmentation, accounting for equivariance and invariance under spatial and photometric transformations.
• To improve generalization and performance by replacing fixed handcrafted strong augmentations with a learnable, adversarial augmentation strategy.
• To demonstrate that semi-supervised learning with selective unlabelled data can surpass fully supervised baselines in surgical segmentation benchmarks.

Proposed method

• Adapts FixMatch’s dual-branch training: one branch with weak augmentations (e.g., flips, rotations) for pseudo-label generation, and another with strong augmentations for consistency regularization.
• Applies spatial transformations (e.g., rotations, flips) with inverse transformation to preserve segmentation mask consistency, ensuring equivariance.
• Introduces an iterative Fast Gradient Sign Method (I-FGSM) as a trainable strong augmentation to generate adversarial examples that guide model robustness.
• Uses confidence thresholding to select high-confidence pixels from weakly augmented outputs as pseudo-labels for training the strongly augmented branch.
• Restricts adversarial perturbations within an ϵ-neighbourhood to preserve instrument integrity while enhancing generalization.
• Trains a shared encoder-decoder backbone on mixed batches of labeled and unlabeled data, minimizing consistency loss between pseudo-labels and strong-augmented predictions.

Figure 1: Representative sample images from Robust-MIS 2019 of laparoscopic surgery (left) and state-of-the-art instrument segmentation results (right). True positive (yellow), true negative (black), false positive (purple), and false negative (red).

Experimental results

Research questions

• RQ1 Can a semi-supervised learning framework adapted from FixMatch achieve superior performance in surgical instrument segmentation compared to fully supervised methods?
• RQ2 How do equivariance and invariance properties of augmentations affect segmentation performance in surgical image analysis?
• RQ3 Does a learnable adversarial augmentation strategy improve model generalization beyond fixed handcrafted augmentations?
• RQ4 What is the optimal balance between labeled and unlabeled data for achieving high segmentation accuracy in low-data regimes?

Key findings

• SegMatch achieved a mean Dice score of 84.9 on Robust-MIS 2019 using only 30% labeled data (1,794 images) and 17,000 unlabelled images, outperforming fully supervised baselines.
• The method surpassed state-of-the-art semi-supervised semantic segmentation models across multiple labeled-to-unlabeled data ratios, particularly at 1:9 and 3:7 ratios.
• I-FGSM with K=50 iterations and ε=0.08 yielded the highest performance (84.9 mean Dice), indicating that iterative adversarial augmentation improves robustness.
• Performance peaked at ε ≈ 0.08; beyond this, further increases degraded performance, confirming a trade-off between perturbation strength and model stability.
• I-FGSM outperformed one-step attacks (e.g., FGSM, C&W) and showed better success rates than FGSM, though it also declined more sharply beyond the optimal ε.
• The use of adversarial augmentations led to a significant improvement in segmentation consistency and generalization, especially in low-data regimes.

Figure 2: SegMatch training process structure. The top row is the fully supervised pathway which follows the traditional segmentation model training process. The two bottom rows form the unsupervised learning pathway, where one branch uses a weakly augmented image fed into the model to compute predi