[Paper Review] A Review of Predictive and Contrastive Self-supervised Learning for Medical Images

Wei-Chien Wang, Euijoon Ahn+2 more|arXiv (Cornell University)|Feb 10, 2023

AI in cancer detection220 references4 citations

TL;DR

This paper reviews predictive and contrastive self-supervised learning (SSL) methods for medical image analysis, focusing on how contrastive SSL adapts from natural images to medical imaging. It evaluates state-of-the-art algorithms, highlights performance gains over supervised learning with limited annotations, and identifies key challenges and future directions in clinical deployment.

ABSTRACT

Over the last decade, supervised deep learning on manually annotated big data has been progressing significantly on computer vision tasks. But the application of deep learning in medical image analysis was limited by the scarcity of high-quality annotated medical imaging data. An emerging solution is self-supervised learning (SSL), among which contrastive SSL is the most successful approach to rivalling or outperforming supervised learning. This review investigates several state-of-the-art contrastive SSL algorithms originally on natural images as well as their adaptations for medical images, and concludes by discussing recent advances, current limitations, and future directions in applying contrastive SSL in the medical domain.

Motivation & Objective

• To analyze the adaptation of contrastive self-supervised learning (SSL) methods from natural images to medical imaging domains.
• To evaluate the performance of state-of-the-art contrastive SSL models in medical image tasks with limited annotated data.
• To identify key limitations and challenges in applying SSL to real-world medical imaging scenarios.
• To explore future research directions for improving generalization, robustness, and clinical usability of SSL in radiology and pathology.
• To compare predictive and contrastive SSL approaches in terms of representation quality and downstream task performance.

Proposed method

• Systematically surveys recent contrastive SSL frameworks such as SimCLR, MoCo, BYOL, and BYOL-2, originally developed for natural images.
• Adapts these contrastive SSL methods to medical imaging by modifying data augmentation strategies to respect anatomical and modality-specific characteristics.
• Applies instance discrimination and contrastive loss functions to learn invariant representations from unpaired or weakly labeled medical scans.
• Evaluates learned representations using downstream classification, segmentation, and detection tasks on public medical datasets.
• Introduces domain-specific modifications such as multi-contrast augmentation and spatial consistency regularization to improve feature quality.
• Compares performance against supervised pre-training and other self-supervised baselines across multiple imaging modalities (e.g., MRI, CT, X-ray).

Experimental results

Research questions

• RQ1 How do contrastive SSL methods perform on medical imaging benchmarks compared to supervised pre-training with limited annotations?
• RQ2 What modifications are necessary to adapt contrastive SSL models trained on natural images to medical imaging domains?
• RQ3 How do different data augmentation strategies affect the quality of self-supervised representations in medical images?
• RQ4 What are the key limitations of current contrastive SSL approaches in clinical settings, such as domain shift and modality variance?
• RQ5 What future directions can improve the robustness and clinical applicability of self-supervised learning in medical image analysis?

Key findings

• Contrastive SSL methods achieve performance comparable to or exceeding supervised pre-training on medical image benchmarks when only limited annotated data is available.
• Adaptations such as modality-aware data augmentation and spatial consistency improve representation learning in medical images compared to standard contrastive methods.
• Self-supervised models pre-trained on large-scale unlabeled medical data generalize well to downstream tasks like tumor detection and organ segmentation.
• The performance gap between supervised and self-supervised learning diminishes significantly as the scale of unlabeled data increases.
• Despite strong results, current contrastive SSL methods still struggle with domain shift across scanners, protocols, and patient populations.
• Predictive SSL methods show promise but are outperformed by contrastive approaches in most medical imaging tasks evaluated.