[Paper Review] ZScribbleSeg: Zen and the Art of Scribble Supervised Medical Image Segmentation

Ke Zhang, Xiahai Zhuang|arXiv (Cornell University)|Jan 12, 2023

Advanced Neural Network Applications4 citations

TL;DR

ZScribbleSeg proposes a novel weakly supervised medical image segmentation framework that leverages efficient scribble annotations and spatial prior regularization to achieve state-of-the-art performance with minimal annotation effort. By simulating diverse scribble patterns via mixup-occlusion and introducing a novel class mixture ratio estimation for spatial prior regularization, ZScribbleSeg significantly improves segmentation accuracy, especially on small datasets and irregular structures, outperforming existing methods on ACDC, MSCMRseg, MyoPS, and PPSS datasets.

ABSTRACT

Curating a large scale fully-annotated dataset can be both labour-intensive and expertise-demanding, especially for medical images. To alleviate this problem, we propose to utilize solely scribble annotations for weakly supervised segmentation. Existing solutions mainly leverage selective losses computed solely on annotated areas and generate pseudo gold standard segmentation by propagating labels to adjacent areas. However, these methods could suffer from the inaccurate and sometimes unrealistic pseudo segmentation due to the insufficient supervision and incomplete shape features. Different from previous efforts, we first investigate the principle of ''good scribble annotations'', which leads to efficient scribble forms via supervision maximization and randomness simulation. Furthermore, we introduce regularization terms to encode the spatial relationship and shape prior, where a new formulation is developed to estimate the mixture ratios of label classes. These ratios are critical in identifying the unlabeled pixels for each class and correcting erroneous predictions, thus the accurate estimation lays the foundation for the incorporation of spatial prior. Finally, we integrate the efficient scribble supervision with the prior into a unified framework, denoted as ZScribbleSeg, and apply the method to multiple scenarios. Leveraging only scribble annotations, ZScribbleSeg set new state-of-the-arts on four segmentation tasks using ACDC, MSCMRseg, MyoPS and PPSS datasets.

Motivation & Objective

• To address the high annotation cost of fully supervised medical image segmentation by enabling training with only scribble-level annotations.
• To improve model robustness and accuracy in weakly supervised segmentation, especially for small datasets and irregularly shaped pathologies.
• To develop a principled approach for generating efficient scribble annotations that maximize supervision with minimal effort.
• To integrate spatial and shape priors into the training process through accurate estimation of class mixture ratios.
• To achieve performance comparable to fully supervised methods using only scribble annotations.

Proposed method

• Proposes a supervision augmentation strategy using mixup and occlusion operations to simulate diverse, random, and wide-range scribble patterns that maximize annotation efficiency.
• Introduces a novel class mixture ratio estimation method to compute the prior distribution of label classes (π), enabling accurate identification of unlabeled pixels per class.
• Develops a spatial prior loss based on marginal probability and spatial energy to regularize predictions and correct shape inaccuracies from incomplete supervision.
• Combines the augmented scribble supervision with spatial prior regularization in a unified framework, ZScribbleSeg, trained via end-to-end deep learning.
• Employs a new spatial energy formulation to estimate the likelihood of unlabeled pixels belonging to each class without relying on model predictions.
• Uses a deep neural network (ZScribbleNet) to jointly optimize segmentation with both supervision augmentation and prior-based regularization.

Experimental results

Research questions

• RQ1 What characteristics define 'good' scribble annotations that maximize supervision efficiency with minimal annotation cost?
• RQ2 How can we effectively augment weak supervision from scribbles without increasing manual labeling effort?
• RQ3 How can spatial and shape priors be incorporated into weakly supervised segmentation to correct inaccurate predictions?
• RQ4 Can accurate estimation of class mixture ratios (π) improve segmentation performance in low-supervision regimes?
• RQ5 Can a unified framework combining supervision augmentation and prior regularization generalize across diverse medical and natural image segmentation tasks?

Key findings

• ZScribbleSeg achieved state-of-the-art performance on four diverse segmentation tasks: ACDC (86.2% ± 8.6% Dice), MSCMRseg (87.0% ± 5.8% Dice), MyoPS (63.6% ± 18.8% Dice), and PPSS (83.8% ± 5.0% Dice).
• On the MyoPS dataset, which involves irregular pathology segmentation with a small training set, ZScribbleSeg outperformed all baselines, including CycleMix and ShapePU, demonstrating robustness in challenging scenarios.
• In edema segmentation on MyoPS, ZScribbleSeg achieved a Hausdorff distance (HD) score comparable to the fully supervised FullSupUNet, while other methods failed due to poor shape estimation.
• The method generalized well to natural scene images (PPSS), achieving Dice scores comparable to FullSupUNet and setting a new SOTA for scribble-supervised segmentation in this domain.
• Visualization results showed that ZScribbleSeg produced less noisy, more realistic segmentations than both other weakly supervised methods and the fully supervised baseline in complex occlusion cases.
• Ablation studies confirmed that both supervision augmentation and spatial prior regularization contribute synergistically to performance, with the class mixture ratio estimation being critical for accurate spatial regularization.