[Paper Review] Improved skin lesion recognition by a Self-Supervised Curricular Deep Learning approach

Kirill Sirotkin, Marcos Escudero-Viñolo+2 more|arXiv (Cornell University)|Dec 22, 2021

Cutaneous Melanoma Detection and Management5 citations

TL;DR

This paper proposes a self-supervised curricular deep learning approach that pretrains convolutional neural networks on unlabeled dermatoscopic images using a sequence of pretext tasks ordered by increasing complexity. The method achieves state-of-the-art performance on the ISIC-2019 skin lesion classification benchmark, outperforming both ImageNet-pretrained models and individual pretext task pretraining by focusing model attention on lesion regions and reducing pretraining time significantly.

ABSTRACT

State-of-the-art deep learning approaches for skin lesion recognition often require pretraining on larger and more varied datasets, to overcome the generalization limitations derived from the reduced size of the skin lesion imaging datasets. ImageNet is often used as the pretraining dataset, but its transferring potential is hindered by the domain gap between the source dataset and the target dermatoscopic scenario. In this work, we introduce a novel pretraining approach that sequentially trains a series of Self-Supervised Learning pretext tasks and only requires the unlabeled skin lesion imaging data. We present a simple methodology to establish an ordering that defines a pretext task curriculum. For the multi-class skin lesion classification problem, and ISIC-2019 dataset, we provide experimental evidence showing that: i) a model pretrained by a curriculum of pretext tasks outperforms models pretrained by individual pretext tasks, and ii) a model pretrained by the optimal pretext task curriculum outperforms a model pretrained on ImageNet. We demonstrate that this performance gain is related to the fact that the curriculum of pretext tasks better focuses the attention of the final model on the skin lesion. Beyond performance improvement, this strategy allows for a large reduction in the training time with respect to ImageNet pretraining, which is especially advantageous for network architectures tailored for a specific problem.

Motivation & Objective

• To address the limited performance of deep learning models on small, specialized medical imaging datasets like ISIC-2019 due to insufficient labeled data.
• To overcome the domain gap between ImageNet and dermatoscopic images by using self-supervised pretraining on target-domain data only.
• To improve model generalization and attention focus on skin lesions by designing an optimal curriculum of self-supervised pretext tasks.
• To reduce computational cost compared to ImageNet pretraining, especially for custom or task-specific network architectures.
• To investigate the impact of pretext task ordering on downstream performance and representation similarity.

Proposed method

• The method employs a sequence of self-supervised pretext tasks—rotation prediction (ODC), relative location, and contrastive learning (MoCo v2)—trained in a curriculum order.
• A curriculum ordering strategy is established based on the downstream classification accuracy of individual pretext tasks, prioritizing those that yield higher performance.
• The model is pretrained on the unlabeled ISIC-2019 dataset using this sequential pretext task pipeline before fine-tuning on the labeled classification task.
• Representation similarity between pretext tasks is analyzed using Central Kernel Alignment (CKA) to assess feature space alignment and inform curriculum design.
• The approach avoids reliance on ImageNet by leveraging only unlabeled dermatoscopic images for pretraining.
• Qualitative analysis uses class activation maps (CAMs) to evaluate attention focus on skin lesions and their borders.

Experimental results

Research questions

• RQ1 Does a curriculum of self-supervised pretext tasks improve skin lesion classification accuracy compared to individual pretext tasks?
• RQ2 Can a self-supervised curriculum on target-domain data outperform ImageNet-supervised pretraining in skin lesion recognition?
• RQ3 How does the order of pretext tasks affect the final model’s performance and attention focus?
• RQ4 To what extent do the learned representations from different pretext tasks align, and how does this affect the curriculum outcome?
• RQ5 Can representation similarity metrics like CKA guide the automatic construction of an optimal pretext task curriculum?

Key findings

• A model pretrained with a curated sequence of pretext tasks achieved 72.8% top-1 accuracy on the ISIC-2019 dataset, surpassing the ImageNet-pretrained baseline.
• The optimal curriculum outperformed models pretrained with individual pretext tasks, demonstrating the benefit of sequential, ordered pretraining.
• The curriculum-based approach improved attention focus on skin lesions, as confirmed by class activation maps showing stronger activation on lesion regions.
• Pretraining on the ISIC-2019 dataset using the proposed curriculum required significantly less time than ImageNet pretraining, especially for custom architectures.
• CKA analysis revealed that ODC and MoCo v2 produced highly similar representations, while Relative Location features were dissimilar, explaining performance differences in curriculum sequences.
• The full curriculum (all three tasks) outperformed mixed and anti-curriculum orderings, though not always the best—indicating that representation similarity and task sequence matter more than just task count.