[Paper Review] Multi-class Semantic Segmentation of Skin Lesions via Fully Convolutional Networks

Manu S. Goyal, Moi Hoon Yap+1 more|arXiv (Cornell University)|Nov 28, 2017

Cutaneous Melanoma Detection and Management20 citations

TL;DR

This paper proposes an end-to-end multi-class semantic segmentation framework for skin lesions using fully convolutional networks (FCNs) with two-tier transfer learning and a hybrid loss function to address class imbalance. The method achieves a Dice score of 78.5% for naevi, 65.3% for melanoma, and 55.7% for seborrhoeic keratosis on the ISIC-2017 test set, with lesion diagnosis accuracy reaching 84.62% for melanoma after post-processing.

ABSTRACT

Melanoma is clinically difficult to distinguish from common benign skin lesions, particularly melanocytic naevus and seborrhoeic keratosis. The dermoscopic appearance of these lesions has huge intra-class variations and high inter-class visual similarities. Most current research is focusing on single-class segmentation irrespective of classes of skin lesions. In this work, we evaluate the performance of deep learning on multi-class segmentation of ISIC-2017 challenge dataset, which consists of 2,750 dermoscopic images. We propose an end-to-end solution using fully convolutional networks (FCNs) for multi-class semantic segmentation to automatically segment the melanoma, seborrhoeic keratosis and naevus. To improve the performance of FCNs, transfer learning and a hybrid loss function are used. We evaluate the performance of the deep learning segmentation methods for multi-class segmentation and lesion diagnosis (with post-processing method) on the testing set of the ISIC-2017 challenge dataset. The results showed that the two-tier level transfer learning FCN-8s achieved the overall best result with \ extit{Dice} score of 78.5% in a naevus category, 65.3% in melanoma, and 55.7% in seborrhoeic keratosis in multi-class segmentation and Accuracy of 84.62% for recognition of melanoma in lesion diagnosis.

Motivation & Objective

• To address the challenge of multi-class semantic segmentation of skin lesions with high intra-class variation and inter-class visual similarity.
• To overcome data scarcity in melanoma and seborrhoeic keratosis categories through two-tier transfer learning.
• To improve segmentation performance on imbalanced datasets using a novel hybrid loss function.
• To evaluate the effectiveness of deep learning models for both multi-class segmentation and lesion diagnosis using post-processing.
• To establish a new baseline for multi-class medical image segmentation in dermatology.

Proposed method

• Utilizes fully convolutional networks (FCNs) including FCN-AlexNet, FCN-32s, FCN-16s, and FCN-8s for pixel-wise classification of three skin lesion types.
• Applies two-tier transfer learning: pre-training on ImageNet followed by fine-tuning on the ISIC-2017 dataset to enhance feature learning.
• Designs a hybrid loss function combining cross-entropy and Dice loss to mitigate class imbalance in the segmentation task.
• Resizes all dermoscopic images to 500×375 pixels to standardize input and reduce computational cost.
• Employs a priority-based post-processing strategy for lesion diagnosis: malignant (melanoma) takes precedence over benign lesions (SK and naevi), with class priority based on training set size.
• Evaluates models on the ISIC-2017 test set using Dice score, MCC, sensitivity, and accuracy metrics.

Experimental results

Research questions

• RQ1 Can fully convolutional networks effectively perform multi-class semantic segmentation of skin lesions with high visual similarity across classes?
• RQ2 How does two-tier transfer learning improve segmentation performance on limited medical imaging datasets?
• RQ3 To what extent does a hybrid loss function mitigate performance degradation due to class imbalance in skin lesion segmentation?
• RQ4 Can post-processing strategies significantly improve lesion diagnosis accuracy despite suboptimal segmentation for certain classes?
• RQ5 What is the performance gap between segmentation accuracy and diagnostic accuracy in multi-class skin lesion analysis?

Key findings

• The FCN-8s model with two-tier transfer learning achieved the highest overall performance, with a Dice score of 78.5% for naevi, 65.3% for melanoma, and 55.7% for seborrhoeic keratosis on the ISIC-2017 test set.
• FCN-AlexNet achieved the best Dice score (0.819) and MCC (0.814) for the naevi class, indicating strong performance for the most frequent class.
• Despite lower segmentation scores, the post-processing method achieved 84.62% accuracy in diagnosing melanoma on the test set, demonstrating the value of diagnostic-level refinement.
• The method achieved 74.44% accuracy for seborrhoeic keratosis diagnosis, highlighting the impact of data scarcity on model generalization.
• The overall diagnosis accuracy across all classes was 80.83%, indicating the robustness of the post-processing strategy in real-world diagnostic scenarios.
• Segmentation performance was significantly lower for melanoma and seborrhoeic keratosis due to limited training samples (521 and 386, respectively), compared to 1,843 naevi images.