[Paper Review] Self-supervised learning for skin cancer diagnosis with limited training data

Hamish Haggerty, Rohitash Chandra|arXiv (Cornell University)|Jan 1, 2024

Cutaneous Melanoma Detection and Management4 citations

TL;DR

This paper proposes self-supervised learning (SSL) using Barlow Twins for skin cancer diagnosis with limited labelled data, demonstrating that SSL pre-training on ImageNet outperforms traditional supervised pre-training (ResNet-50) by achieving 70% test accuracy versus 66%. Further gains are achieved by fine-tuning after a second SSL pre-training on unlabelled skin lesion data, offering a scalable alternative to costly data annotation.

ABSTRACT

Early cancer detection is crucial for prognosis, but many cancer types lack large labelled datasets required for developing deep learning models. This paper investigates self-supervised learning (SSL) as an alternative to the standard supervised pre-training on ImageNet for scenarios with limited training data using a deep learning model (ResNet-50). We first demonstrate that SSL pre-training on ImageNet (via the Barlow Twins SSL algorithm) outperforms supervised pre-training (SL) using a skin lesion dataset with limited training samples. We then consider extit{further} SSL pre-training (of the two ImageNet pre-trained models) on task-specific datasets, where our implementation is motivated by supervised transfer learning. This approach significantly enhances initially SL pre-trained models, closing the performance gap with initially SSL pre-trained ones. Surprisingly, further pre-training on just the limited fine-tuning data achieves this performance equivalence. Linear probe experiments reveal that improvement stems from enhanced feature extraction. Hence, we find that minimal further SSL pre-training on task-specific data can be as effective as large-scale SSL pre-training on ImageNet for medical image classification tasks with limited labelled data. We validate these results on an oral cancer histopathology dataset, suggesting broader applicability across medical imaging domains facing labelled data scarcity.

Motivation & Objective

• Address the challenge of limited and imbalanced labelled medical image datasets in cancer diagnosis, particularly for rare or underdiagnosed cancers.
• Investigate whether self-supervised pre-training can outperform conventional supervised transfer learning in low-data regimes for skin cancer classification.
• Explore the impact of additional self-supervised pre-training on unlabelled target-domain data (skin lesions) to boost performance without further annotation.
• Evaluate the effectiveness of SSL versus supervised pre-training across different model architectures and data scales.
• Provide a scalable, data-efficient framework applicable to other low-resource medical image classification tasks.

Proposed method

• Pre-train a ResNet-50 backbone using the Barlow Twins contrastive self-supervised learning algorithm on ImageNet to learn generalizable features without labels.
• Fine-tune the pre-trained SSL model on a small, labelled skin lesion dataset (ISIC) for binary or multi-class skin cancer classification.
• Compare performance against a standard supervised transfer learning baseline, where the same ResNet-50 is pre-trained on ImageNet with supervised labels.
• Perform a second self-supervised pre-training step on unlabelled skin lesion images using Barlow Twins to further improve feature representation before final fine-tuning.
• Use standard data augmentation strategies (e.g., crop, color jitter) during SSL pre-training, selected based on known effectiveness in prior SSL work.
• Evaluate models on a large, held-out test set using standard metrics: accuracy and balanced accuracy.

Figure 1: A residual block [ 24 ]

Experimental results

Research questions

• RQ1 Does self-supervised pre-training with Barlow Twins outperform supervised pre-training on ImageNet for skin cancer diagnosis under limited labelled data?
• RQ2 Can additional self-supervised pre-training on unlabelled skin lesion images further improve classification performance beyond ImageNet pre-training?
• RQ3 Does increasing model size (e.g., beyond ResNet-50) improve transfer performance when pre-trained via SSL on ImageNet?
• RQ4 How does SSL transfer performance compare to supervised transfer in the context of class-imbalanced medical datasets?
• RQ5 Can the proposed SSL framework be generalized to other low-resource medical image classification problems?

Key findings

• Self-supervised pre-training with Barlow Twins achieved a mean test accuracy of 70% on the ISIC skin lesion dataset, outperforming the supervised transfer baseline of 66%.
• A second round of self-supervised pre-training on unlabelled skin lesion images led to further performance gains, suggesting that collecting more unlabelled data is a viable alternative to costly labelling.
• Pre-training a larger model (with over 10x more parameters than ResNet-50) on ImageNet via SSL did not improve transfer performance, indicating that model scale alone does not enhance SSL transfer in this setup.
• The performance gap between SSL and supervised transfer was substantial, with SSL showing a 4 percentage point improvement in accuracy under low-data conditions.
• The results suggest that SSL transfer is a superior strategy to supervised transfer in low-labelled-data regimes, especially when combined with target-domain self-supervision.
• The framework is generalizable and applicable to other medical image classification tasks with limited annotations, such as oral or pancreatic cancer diagnosis.

Figure 2: Joint embedding architecture