[Paper Review] A Deep Multi-task Learning Approach to Skin Lesion Classification

Haofu Liao, Jiebo Luo|arXiv (Cornell University)|Dec 9, 2018

Cutaneous Melanoma Detection and ManagementMedicine28 references3 citations

TL;DR

This paper proposes a deep multi-task learning framework that jointly optimizes skin lesion classification and body location classification using a ResNet-50 backbone with specialized loss functions. By leveraging the anatomical distribution patterns of skin lesions as an inductive bias, the method achieves a 0.80 mean average precision (mAP) in universal skin lesion classification, outperforming single-task baselines and improving both lesion and body location classification performance.

ABSTRACT

Skin lesion identification is a key step toward dermatological diagnosis. When describing a skin lesion, it is very important to note its body site distribution as many skin diseases commonly affect particular parts of the body. To exploit the correlation between skin lesions and their body site distributions, in this study, we investigate the possibility of improving skin lesion classification using the additional context information provided by body location. Specifically, we build a deep multi-task learning (MTL) framework to jointly optimize skin lesion classification and body location classification (the latter is used as an inductive bias). Our MTL framework uses the state-of-the-art ImageNet pretrained model with specialized loss functions for the two related tasks. Our experiments show that the proposed MTL based method performs more robustly than its standalone (single-task) counterpart.

Motivation & Objective

• To address the challenge of universal skin lesion classification across diverse lesion types, which is difficult with traditional handcrafted features.
• To investigate whether incorporating body site distribution as an auxiliary task improves skin lesion classification performance.
• To develop a deep learning framework that jointly learns lesion type and anatomical location using a shared CNN backbone.
• To build a large-scale, publicly available skin lesion dataset (21,657 images) from DermQuest for training and evaluation.
• To evaluate the generalization and interpretability of the model through image retrieval and attention visualization.

Proposed method

• The method employs a deep multi-task learning framework based on a pre-trained ResNet-50 model fine-tuned on skin lesion and body location classification.
• Two separate fully connected (FC) layers are used for the primary task (skin lesion classification) and the auxiliary task (body location classification), each with its own specialized loss function.
• Feature representations are extracted from the final pooling layer (pool5) for image retrieval and from the final convolutional layer (res5c) for attention map generation.
• Image attention is computed by taking the weighted average of activation maps using the learned FC layer weights, producing class-specific attention maps for both tasks.
• An ensemble of the multi-task model and its single-task counterpart is used to boost performance, achieving the highest mAP.
• The model is trained end-to-end with a joint optimization objective that encourages shared feature learning while maintaining task-specific representations.

Experimental results

Research questions

• RQ1 Can joint learning of skin lesion classification and body location classification improve the performance of universal skin lesion classification?
• RQ2 Does the anatomical distribution of skin lesions serve as a useful inductive bias for improving lesion classification via multi-task learning?
• RQ3 How does the performance of the multi-task model compare to its single-task counterparts in both lesion and body location classification?
• RQ4 To what extent do attention maps reflect meaningful lesion and anatomical region localization in dermatological images?
• RQ5 Can image retrieval using learned features from the multi-task model retrieve semantically similar lesions and body parts?

Key findings

• The proposed multi-task learning model achieves a mean average precision (mAP) of 0.80 in universal skin lesion classification, outperforming single-task baselines.
• The body location classification task also sees improved performance under the multi-task setting, with higher top-1 and top-3 accuracy compared to training on body location alone.
• Image retrieval results show that the model retrieves visually and semantically similar images, with correctly matched queries framed in green and mismatches in red.
• Attention visualization confirms that the model learns to focus on lesion regions for the primary task and on anatomical body parts for the auxiliary task, though some class-specific biases (e.g., 'alopecia') show overlapping attention maps.
• The ensemble of the multi-task model and its single-task counterpart achieves the highest mAP, indicating complementary performance gains.
• The study confirms that body site predilection is a valuable inductive bias that enhances feature learning and generalization in skin lesion classification.