[Paper Review] Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)

David Gutman, Noel Codella+5 more|arXiv (Cornell University)|May 4, 2016

Cutaneous Melanoma Detection and Management364 citations

TL;DR

Describes a public, multi-task melanoma diagnosis benchmark (segmentation, dermoscopic feature detection, and classification) using ISIC dermoscopy images with 900 training and 379 test images, detailing tasks, evaluation metrics, and results from 38 participants.

ABSTRACT

In this article, we describe the design and implementation of a publicly accessible dermatology image analysis benchmark challenge. The goal of the challenge is to sup- port research and development of algorithms for automated diagnosis of melanoma, a lethal form of skin cancer, from dermoscopic images. The challenge was divided into sub-challenges for each task involved in image analysis, including lesion segmentation, dermoscopic feature detection within a lesion, and classification of melanoma. Training data included 900 images. A separate test dataset of 379 images was provided to measure resultant performance of systems developed with the training data. Ground truth for both training and test sets was generated by a panel of dermoscopic experts. In total, there were 79 submissions from a group of 38 participants, making this the largest standardized and comparative study for melanoma diagnosis in dermoscopic images to date. While the official challenge duration and ranking of participants has concluded, the datasets remain available for further research and development.

Motivation & Objective

• Provide a publicly accessible benchmark to advance automated melanoma diagnosis from dermoscopic images.
• Evaluate segmentation, dermoscopic feature detection, and malignant classification in a standardized framework.
• Enable cross-institutional comparison of algorithms using expert-ground-truth annotations.

Proposed method

• Three challenge parts map to lesion analysis: segmentation, dermoscopic feature detection/segmentation (globules and streaks using superpixels), and disease classification.
• Training data: 900 images with ground truth; test set: 379 images.
• Ground truth annotated by dermoscopic experts.
• Evaluation platform Covalic enabled real-time submissions and ranking.
• Metrics include pixel-level accuracy, sensitivity, specificity, Dice, and Jaccard for segmentation; accuracy, sensitivity, specificity, average precision, AUC, SP95, SP98 for classification.

Experimental results

Research questions

• RQ1 How well can automated methods segment dermoscopic skin lesions compared with expert annotations?
• RQ2 Can automated systems accurately detect dermoscopic features (globules, streaks) within lesions, and how do superpixel-based labels affect performance?
• RQ3 What is the performance of automated melanoma classification (benign vs malignant) on held-out test data compared with expert benchmarks?

Key findings

• 79 submissions from 38 participants across five task variants.
• Segmentation top: AC 0.953, SE 0.910, SP 0.965, DI 0.910, JA 0.843 (Part 1).
• Segmentation (Part 2B) top: AC 0.962, SE 0.396, SP 0.968, DI 0.128, JA 0.070.
• Classification top: Part 2: AC 0.916, SE 0.505, SP 0.920, AP 0.243, AUC 0.677.
• Classification Part 3: AC 0.855, SE 0.507, SP 0.941, AP 0.637, AUC 0.804.
• Classification Part 3B: AC 0.855, SE 0.547, SP 0.931, AP 0.624, AUC 0.783.”],
• table_headers([
• Part
• AC
• SE
• SP
• AP
• AUC
• SP95
• SP98