[Paper Review] Surrogate Supervision for Medical Image Analysis: Effective Deep Learning From Limited Quantities of Labeled Data

Nima Tajbakhsh, Yufei Hu+7 more|arXiv (Cornell University)|Jan 25, 2019

Radiomics and Machine Learning in Medical Imaging17 references12 citations

TL;DR

This paper proposes surrogate supervision as a method to pre-train deep neural networks for medical image analysis using abundant unlabeled data, assigning artificial labels via rotation, reconstruction, or colorization. It demonstrates that pre-training with surrogate supervision significantly improves performance on small-labeled datasets—especially for 3D medical imaging—outperforming both random initialization and transfer learning from natural images.

ABSTRACT

We investigate the effectiveness of a simple solution to the common problem of deep learning in medical image analysis with limited quantities of labeled training data. The underlying idea is to assign artificial labels to abundantly available unlabeled medical images and, through a process known as surrogate supervision, pre-train a deep neural network model for the target medical image analysis task lacking sufficient labeled training data. In particular, we employ 3 surrogate supervision schemes, namely rotation, reconstruction, and colorization, in 4 different medical imaging applications representing classification and segmentation for both 2D and 3D medical images. 3 key findings emerge from our research: 1) pre-training with surrogate supervision is effective for small training sets; 2) deep models trained from initial weights pre-trained through surrogate supervision outperform the same models when trained from scratch, suggesting that pre-training with surrogate supervision should be considered prior to training any deep 3D models; 3) pre-training models in the medical domain with surrogate supervision is more effective than transfer learning from an unrelated domain (e.g., natural images), indicating the practical value of abundant unlabeled medical image data.

Motivation & Objective

• To address the challenge of limited labeled data in medical image analysis, particularly for 3D imaging where transfer learning is ineffective.
• To investigate whether pre-training with surrogate supervision in the medical domain improves model performance compared to random initialization or transfer from natural images.
• To evaluate the effectiveness of surrogate supervision across diverse medical imaging tasks, including classification and segmentation in both 2D and 3D modalities.
• To determine whether surrogate supervision provides better weight initialization than transfer learning when labeled data is scarce.

Proposed method

• Pre-train deep neural networks using artificial labels generated from unlabeled medical images via three surrogate supervision schemes: rotation prediction, image reconstruction, and colorization.
• Assign surrogate labels to unlabeled images—e.g., predicting rotation angle, reconstructing original image from corrupted version, or predicting color from grayscale input.
• Train models on the target medical task using only a fraction (10%, 25%, 50%, 100%) of the labeled data, initialized with weights from surrogate pre-training.
• Compare performance against models trained from random initialization and models fine-tuned from ImageNet pre-trained weights.
• Use standard evaluation metrics: FROC for nodule detection, Dice score for segmentation, and Kappa statistic for classification.
• Apply data splits that preserve disjoint training, validation, and test sets, merging labeled and unlabeled data for surrogate pre-training while removing target task labels from the unlabeled set.

Experimental results

Research questions

• RQ1 Does surrogate supervision improve deep learning performance in medical image analysis when labeled data is limited?
• RQ2 Is pre-training with surrogate supervision in the medical domain more effective than transfer learning from ImageNet?
• RQ3 How does surrogate supervision impact model performance as the size of the labeled training set increases?
• RQ4 Can surrogate supervision serve as a viable alternative to training from scratch for 3D medical image analysis?

Key findings

• Pre-training with surrogate supervision consistently improves model performance over training from random initialization, especially when only 10% or 25% of the labeled data is used.
• For 3D medical imaging tasks like lung lobe segmentation, surrogate supervision significantly outperforms training from scratch, indicating its value for 3D architectures commonly trained from scratch.
• Surrogate supervision in the medical domain is more effective than transfer learning from ImageNet, particularly for tasks involving fundus or CT images that differ substantially from natural images.
• The performance gain from surrogate supervision diminishes as the labeled training set size increases, suggesting it is most beneficial for small-data scenarios.
• For skin segmentation, which is closer to natural images in domain, ImageNet transfer still outperforms surrogate supervision, but surrogate pre-training remains superior to random initialization.
• The results demonstrate that surrogate supervision provides more effective weight initialization than either random weights or unrelated-domain pre-training, especially in low-data regimes.