[Paper Review] Going to Extremes: Weakly Supervised Medical Image Segmentation

Holger R. Roth, Dong Yang+3 more|arXiv (Cornell University)|Sep 25, 2020

Advanced Neural Network Applications3 references4 citations

TL;DR

This paper proposes a weakly supervised 3D medical image segmentation method that uses only extreme point clicks—one per anatomical dimension—as minimal user input. It leverages the random walker algorithm to generate an initial noisy segmentation, which is then refined via iterative training of a fully convolutional network (FCN) enhanced with a point-based loss and attention mechanism, achieving performance close to fully supervised methods on six diverse medical imaging tasks, including surpassing fully supervised results on the MO-Gallbladder dataset.

ABSTRACT

Medical image annotation is a major hurdle for developing precise and robust machine learning models. Annotation is expensive, time-consuming, and often requires expert knowledge, particularly in the medical field. Here, we suggest using minimal user interaction in the form of extreme point clicks to train a segmentation model which, in effect, can be used to speed up medical image annotation. An initial segmentation is generated based on the extreme points utilizing the random walker algorithm. This initial segmentation is then used as a noisy supervision signal to train a fully convolutional network that can segment the organ of interest, based on the provided user clicks. Through experimentation on several medical imaging datasets, we show that the predictions of the network can be refined using several rounds of training with the prediction from the same weakly annotated data. Further improvements are shown utilizing the clicked points within a custom-designed loss and attention mechanism. Our approach has the potential to speed up the process of generating new training datasets for the development of new machine learning and deep learning-based models for, but not exclusively, medical image analysis.

Motivation & Objective

• To reduce the cost and complexity of medical image annotation by minimizing user interaction to just extreme point clicks on organ boundaries.
• To develop a weakly supervised learning framework that leverages minimal user input to train accurate 3D segmentation models.
• To improve segmentation performance using iterative refinement of predictions from weakly annotated data.
• To integrate user-provided extreme points into the loss function and attention mechanism to guide network learning without increasing labeling burden.
• To evaluate the method across diverse 3D medical imaging tasks and compare its performance to fully supervised and prior weakly supervised approaches.

Proposed method

• The method begins with user-defined extreme point clicks—one per spatial dimension (e.g., leftmost, rightmost, anterior, posterior, superior, inferior) on a 3D medical image.
• An initial segmentation is generated using the random walker algorithm applied to the extreme points, producing a noisy pseudo-label.
• This pseudo-label is used to supervise the training of a fully convolutional network (FCN) for organ segmentation.
• The network is trained iteratively, using its own predictions from the previous round as the new weak supervision signal.
• A novel point-based loss function incorporates the spatial coordinates of the extreme points directly into the loss to guide the network.
• An attention mechanism is introduced that uses the extreme point locations to modulate feature maps, enhancing focus on relevant regions.

Experimental results

Research questions

• RQ1 Can extreme point clicks alone, without dense annotations, be sufficient to train a high-performing 3D medical image segmentation model?
• RQ2 How does iterative refinement using the model’s own predictions improve segmentation performance over successive training rounds?
• RQ3 To what extent does integrating extreme point information into the loss function and attention mechanism enhance model performance compared to standard weakly supervised baselines?
• RQ4 Does the proposed method achieve performance comparable to or exceeding fully supervised training in real-world medical imaging tasks?
• RQ5 How robust is the method across diverse anatomical structures with varying shapes, textures, and complexities?

Key findings

• The proposed method achieved performance close to fully supervised training on four out of six medical imaging tasks, demonstrating strong generalization from minimal user input.
• On the MO-Gallbladder dataset, the weakly supervised model outperformed the fully supervised baseline, achieving a Dice score of 0.874 compared to the fully supervised model’s 0.862.
• The addition of both point-based loss and attention mechanism improved performance in four out of six tasks, with the highest gains observed in complex or irregularly shaped organs.
• The iterative training scheme consistently improved segmentation accuracy across all datasets, with performance increasing over multiple rounds using the same weakly annotated data.
• The random walker initialization provided a robust starting point for segmentation, even in cases with high texture variability or complex geometry, such as the pancreas.
• The method significantly reduces annotation cost by requiring only six point clicks per 3D volume, compared to dense scribbles or full segmentation.