[Paper Review] Revisiting 3D Medical Scribble Supervision: Benchmarking Beyond Cardiac Segmentation

Karol Gotkowski, Klaus Maier‐Hein+1 more|arXiv (Cornell University)|Mar 19, 2024

Medical Imaging and Analysis4 citations

TL;DR

This paper proposes partial losses—a simple yet effective method for 3D medical image segmentation using scribble annotations—by computing loss only on annotated voxels, enabling seamless integration into state-of-the-art models like nnU-Net. The approach achieves state-of-the-art performance across diverse datasets, outperforming both systemic and lightweight methods while demonstrating robust generalization across anatomies, modalities, and scribble styles.

ABSTRACT

Scribble supervision has emerged as a promising approach for reducing annotation costs in medical 3D segmentation by leveraging sparse annotations instead of voxel-wise labels. While existing methods report strong performance, a closer analysis reveals that the majority of research is confined to the cardiac domain, predominantly using ACDC and MSCMR datasets. This over-specialization has resulted in severe overfitting, misleading claims of performance improvements, and a lack of generalization across broader segmentation tasks. In this work, we formulate a set of key requirements for practical scribble supervision and introduce ScribbleBench, a comprehensive benchmark spanning over seven diverse medical imaging datasets, to systematically evaluate the fulfillment of these requirements. Consequently, we uncover a general failure of methods to generalize across tasks and that many widely used novelties degrade performance outside of the cardiac domain, whereas simpler overlooked approaches achieve superior generalization. Finally, we raise awareness for a strong yet overlooked baseline, nnU-Net coupled with a partial loss, which consistently outperforms specialized methods across a diverse range of tasks. By identifying fundamental limitations in existing research and establishing a new benchmark-driven evaluation standard, this work aims to steer scribble supervision toward more practical, robust, and generalizable methodologies for medical image segmentation.

Motivation & Objective

• Address the lack of generalization in existing scribble-supervised methods, which are often tightly coupled to specific segmentation architectures and fail to generalize across medical imaging tasks.
• Overcome the limitation of sparse annotation evaluation being restricted to single datasets, such as cardiac segmentation, by establishing a comprehensive benchmark across diverse anatomical structures and imaging modalities.
• Enable seamless integration of scribble supervision into state-of-the-art segmentation frameworks by decoupling the learning mechanism from the model architecture.
• Demonstrate that a simple, modular approach—partial losses—can surpass complex, task-specific systemic methods without requiring architectural modifications.

Proposed method

• Introduce partial losses, a generalization of partial Cross-Entropy, where loss is computed only on annotated voxels, enabling training from sparse scribbles while preserving the original loss formulation of the segmentation model.
• Extend partial losses beyond Cross-Entropy to other common segmentation losses, such as Dice loss, to maintain compatibility with modern segmentation frameworks like nnU-Net.
• Design a comprehensive benchmark with seven diverse 3D medical imaging datasets covering different organs, pathologies, and imaging modalities (e.g., MRI, CT) to evaluate generalization across the medical domain.
• Systematically compare partial losses against existing systemic and lightweight scribble methods across all datasets, using consistent evaluation protocols and annotation budgets.
• Evaluate robustness across different scribble styles and annotation strategies (e.g., scribbles, slice-level dense annotations, random image sampling) under fixed annotation budgets.
• Demonstrate portability by applying partial losses to a residual U-Net architecture, confirming consistent performance gains independent of the backbone model.

Figure 1: (A): Depiction of the annotation effort reducing benefits of scribble-supervised learning. (B): Partial losses only consider annotated voxels during the loss computation. This idea, initially introduced for the Cross-Entropy loss, can be extended to other popular segmentation losses like t

Experimental results

Research questions

• RQ1 Can a simple, modular loss formulation—partial losses—achieve state-of-the-art performance in 3D medical image segmentation using scribble annotations without architectural modifications?
• RQ2 How does the performance of partial losses generalize across diverse anatomical structures, pathologies, and imaging modalities compared to existing systemic and lightweight scribble methods?
• RQ3 Does the use of partial losses maintain consistent performance across different scribble styles and annotation strategies under a fixed annotation budget?
• RQ4 Can partial losses be effectively transferred to different state-of-the-art segmentation architectures beyond the base nnU-Net model?
• RQ5 Why have previous scribble-supervised methods failed to gain traction in practice, and can a systematic benchmark and modular approach resolve these issues?

Key findings

• Partial losses outperform a pCE-only baseline by a margin of 0.045 in mean Dice score across the benchmark, demonstrating significant performance gains from a simple modification.
• The proposed method achieves state-of-the-art performance across all seven datasets in the benchmark, surpassing both systemic and lightweight methods in generalization and consistency.
• Partial losses show robust generalization across diverse anatomical regions (e.g., heart, liver, brain, kidneys), pathologies (e.g., tumors, atrophy), and imaging modalities (MRI, CT), with consistent performance across all but BraTS, where fuzzy boundaries challenge all methods.
• Among sparse annotation strategies, scribbles outperform slice-level dense annotations under the same annotation budget, and dense annotations underperform due to fewer annotated images, highlighting the efficiency of sparse labeling.
• Qualitative analysis confirms that partial losses produce the most consistent and accurate segmentation boundaries compared to systemic and lightweight methods, especially in complex anatomical regions.
• The method maintains superior performance when transferred to a residual U-Net architecture, confirming its modularity and broad applicability to modern segmentation frameworks.

Figure 2: Examples of segmentation types with (A) depicting a dense segmentation and (B) a scribble segmentation generated by our automatic scribble generation method based on the dense segmentation.