[Paper Review] Multi-task Explainable Skin Lesion Classification

Mahapara Khurshid, Mayank Vatsa+1 more|arXiv (Cornell University)|Oct 11, 2023

Cutaneous Melanoma Detection and ManagementMedicine3 citations

TL;DR

This paper proposes a multi-task, few-shot learning framework that fuses semantic segmentation and classification for explainable skin lesion diagnosis. By leveraging a shared backbone with joint segmentation and classification heads, and using a weighted loss combining both tasks, the model achieves state-of-the-art accuracy (up to 77.64% on 5-shot 2-way classification) while providing attention-based explanations via Grad-CAM, demonstrating strong generalization across dermatoscopic datasets including HAM10000, PH2, and Derm7pt.

ABSTRACT

Skin cancer is one of the deadliest diseases and has a high mortality rate if left untreated. The diagnosis generally starts with visual screening and is followed by a biopsy or histopathological examination. Early detection can aid in lowering mortality rates. Visual screening can be limited by the experience of the doctor. Due to the long tail distribution of dermatological datasets and significant intra-variability between classes, automatic classification utilizing computer-aided methods becomes challenging. In this work, we propose a multitask few-shot-based approach for skin lesions that generalizes well with few labelled data to address the small sample space challenge. The proposed approach comprises a fusion of a segmentation network that acts as an attention module and classification network. The output of the segmentation network helps to focus on the most discriminatory features while making a decision by the classification network. To further enhance the classification performance, we have combined segmentation and classification loss in a weighted manner. We have also included the visualization results that explain the decisions made by the algorithm. Three dermatological datasets are used to evaluate the proposed method thoroughly. We also conducted cross-database experiments to ensure that the proposed approach is generalizable across similar datasets. Experimental results demonstrate the efficacy of the proposed work.

Motivation & Objective

• To address the challenge of limited annotated skin lesion data, especially for rare classes, by developing a few-shot learning approach that generalizes well with minimal labeled examples.
• To improve classification robustness by integrating a segmentation network as an attention mechanism to focus on lesion regions and suppress artifacts like hair and scales.
• To enhance model interpretability by generating Grad-CAM visualizations that explain the decision-making process for medical professionals.
• To evaluate generalization across diverse dermatoscopic datasets (HAM10000, PH2, Derm7pt) through cross-database experiments.
• To optimize the trade-off between segmentation and classification losses using a learnable weighting hyperparameter (λ) for improved performance.

Proposed method

• The framework uses a multi-task learning architecture with a shared feature extractor (ResNet50) and two parallel heads: one for lesion segmentation and one for classification.
• The segmentation output acts as a spatial attention mask, guiding the classification network to focus on the lesion region and reduce interference from artifacts.
• A weighted multi-task loss combines cross-entropy for classification and binary cross-entropy for segmentation, with λ controlling the balance between the two losses.
• The classification head employs a prototypical network with metric learning, computing class prototypes as mean embeddings of support samples and classifying query images via cosine or Euclidean distance.
• Explainability is achieved through Grad-CAM, which highlights discriminative regions in the input image that influenced the model’s prediction.
• The model is trained and evaluated using few-shot episodes (1, 3, 5 shots) across 1000 random episodes, with confidence intervals reported at 75%, 90%, and 95%.

Figure 1: Showcasing the skin samples from HAM10000 (top row), PH2 (middle row) and Derm7pt (bottom row). This figure highlights the need for segmentation in skin lesion classification, as there are artifacts that need to be removed before classifying the image.

Experimental results

Research questions

• RQ1 Can a multi-task learning framework that jointly optimizes segmentation and classification improve few-shot skin lesion classification performance on small datasets?
• RQ2 Does using segmentation as an attention mechanism enhance the model’s ability to focus on relevant lesion features and reduce misclassification due to artifacts?
• RQ3 How does the choice of backbone architecture (e.g., VGG16, ResNet18, DenseNet-121, ResNet50) affect few-shot classification accuracy in skin lesion classification?
• RQ4 What is the optimal trade-off between segmentation and classification loss (via λ) for maximizing classification performance under low-data regimes?
• RQ5 To what extent does the proposed method generalize across different dermatoscopic datasets (HAM10000, PH2, Derm7pt) in cross-database evaluation?

Key findings

• The proposed method achieved an average accuracy of 77.57% on HAM10000, 76.06% on PH2, and 77.64% on Derm7pt for 5-shot 2-way classification, with 95% confidence intervals of ±0.73, ±0.98, and ±0.96, respectively.
• Using ResNet50 as the backbone yielded the highest classification accuracy (77.57% on HAM10000), outperforming VGG16 (73.79%), ResNet18 (76.50%), and DenseNet-121 (74.21%) in the 5-shot setting.
• The optimal λ value for the combined loss was found to be 2, achieving 78.21% accuracy on HAM10000, 75.85% on PH2, and 77.83% on Derm7pt, with higher values (e.g., λ=5) leading to performance degradation due to overfitting.
• Cross-database experiments confirmed the model’s strong generalization capability, with consistent performance across all three datasets, indicating robustness to domain shifts.
• Grad-CAM visualizations demonstrated that the model correctly highlights lesion regions as the primary decision-making features, enhancing interpretability and trust for clinical users.
• Ablation studies confirmed that the segmentation head acts as an effective attention module, improving classification by filtering out irrelevant image regions and artifacts.

Figure 2: Illustrating the basic working of prototypical networks