[Paper Review] Deep Transfer Learning for Automated Diagnosis of Skin Lesions from Photographs

Emma Rocheteau, Doyoon Kim|arXiv (Cornell University)|Nov 6, 2020

Cutaneous Melanoma Detection and Management35 references4 citations

TL;DR

This study evaluates transfer learning with 11 deep neural networks for automated melanoma detection from skin lesion photographs, using ImageNet-pretrained models fine-tuned on the ISIC 2020 dataset. EfficientNet with transfer learning achieved the highest performance, attaining an AUROC of 0.975 ± 0.002 and AUPRC of 0.906 ± 0.006, outperforming both general practitioners (AUROC 0.83 ± 0.03) and dermatologists (AUROC 0.91 ± 0.02).

ABSTRACT

Melanoma is not the most common form of skin cancer, but it is the most deadly. Currently, the disease is diagnosed by expert dermatologists, which is costly and requires timely access to medical treatment. Recent advances in deep learning have the potential to improve diagnostic performance, expedite urgent referrals and reduce burden on clinicians. Through smart phones, the technology could reach people who would not normally have access to such healthcare services, e.g. in remote parts of the world, due to financial constraints or in 2020, COVID-19 cancellations. To this end, we have investigated various transfer learning approaches by leveraging model parameters pre-trained on ImageNet with finetuning on melanoma detection. We compare EfficientNet, MnasNet, MobileNet, DenseNet, SqueezeNet, ShuffleNet, GoogleNet, ResNet, ResNeXt, VGG and a simple CNN with and without transfer learning. We find the mobile network, EfficientNet (with transfer learning) achieves the best mean performance with an area under the receiver operating characteristic curve (AUROC) of 0.931$\pm$0.005 and an area under the precision recall curve (AUPRC) of 0.840$\pm$0.010. This is significantly better than general practitioners (0.83$\pm$0.03 AUROC) and dermatologists (0.91$\pm$0.02 AUROC).

Motivation & Objective

• To evaluate the effectiveness of transfer learning using ImageNet-pretrained models for automated melanoma diagnosis from smartphone photographs.
• To compare the performance of 11 deep learning architectures—ranging from MobileNet to EfficientNet—on the binary classification of melanoma versus benign skin lesions.
• To assess whether transfer learning can improve diagnostic accuracy beyond that of general practitioners and dermatologists using real-world clinical data.
• To investigate model interpretability through integrated gradients to visualize feature importance in lesion images.
• To explore the feasibility of deploying high-accuracy, lightweight models on mobile devices for decentralized, privacy-preserving skin cancer screening.

Proposed method

• Transfer learning was applied by initializing models with ImageNet-pretrained weights and fine-tuning on the ISIC 2020 dataset, which includes 37,648 skin lesion images with a 2% melanoma prevalence.
• A 5-layer standard CNN was trained without transfer learning as a baseline for comparison.
• Data augmentation was applied during training, including random rotations, flips, brightness and saturation shifts, to improve robustness and generalization.
• The integrated gradients method was used to compute feature attributions, calculating importance scores by integrating gradients along a path from a black baseline to the input image.
• Model performance was evaluated using AUROC, AUPRC, precision, and recall, with statistical significance testing across all metrics.
• Static patient features (age, gender, body site) were processed separately and concatenated with the CNN feature output before final prediction.

Experimental results

Research questions

• RQ1 Does transfer learning with ImageNet-pretrained models significantly improve melanoma detection performance compared to training from scratch?
• RQ2 Which deep neural network architecture achieves the highest diagnostic performance on the ISIC 2020 skin lesion classification task?
• RQ3 Can transfer learning-based models outperform human experts—specifically general practitioners and dermatologists—in classifying melanoma from benign lesions?
• RQ4 How do the models focus on relevant features in skin lesion images, and do attention patterns align with clinical diagnostic cues such as irregular borders?
• RQ5 Can lightweight, mobile-optimized models like EfficientNet and MnasNet achieve high performance while enabling on-device inference for privacy compliance?

Key findings

• All 11 deep learning models, including MobileNet and EfficientNet, achieved significantly better performance with transfer learning than without, across all evaluation metrics.
• EfficientNet with transfer learning achieved the highest AUROC of 0.975 ± 0.002, surpassing dermatologists (0.91 ± 0.02) and general practitioners (0.83 ± 0.03).
• The same model achieved an AUPRC of 0.906 ± 0.006, indicating strong precision-recall balance in detecting rare melanoma cases.
• Integrated gradient visualizations revealed that models primarily focus on lesion edges and internal texture, aligning with clinical signs of melanoma such as irregular borders and uneven pigmentation.
• MnasNet and EfficientNet, designed for mobile deployment, achieved the highest performance, suggesting suitability for on-device, privacy-preserving skin cancer screening applications.
• No model was significantly harmed by transfer learning, and all benefited from pre-training, confirming the robust advantage of transfer learning in this medical imaging task.