[Paper Review] Estimating Skin Tone and Effects on Classification Performance in Dermatology Datasets

Newton Mwai Kinyanjui, Timothy Odonga+5 more|arXiv (Cornell University)|Oct 29, 2019

Cutaneous Melanoma Detection and Management30 references37 citations

TL;DR

The paper estimates skin-tone distributions in two dermatology benchmarks using ITA, examines representation gaps, and evaluates classifier performance across skin-tone categories, finding under-representation of darker tones but no clear performance bias.

ABSTRACT

Recent advances in computer vision and deep learning have led to breakthroughs in the development of automated skin image analysis. In particular, skin cancer classification models have achieved performance higher than trained expert dermatologists. However, no attempt has been made to evaluate the consistency in performance of machine learning models across populations with varying skin tones. In this paper, we present an approach to estimate skin tone in benchmark skin disease datasets, and investigate whether model performance is dependent on this measure. Specifically, we use individual typology angle (ITA) to approximate skin tone in dermatology datasets. We look at the distribution of ITA values to better understand skin color representation in two benchmark datasets: 1) the ISIC 2018 Challenge dataset, a collection of dermoscopic images of skin lesions for the detection of skin cancer, and 2) the SD-198 dataset, a collection of clinical images capturing a wide variety of skin diseases. To estimate ITA, we first develop segmentation models to isolate non-diseased areas of skin. We find that the majority of the data in the the two datasets have ITA values between 34.5° and 48°, which are associated with lighter skin, and is consistent with under-representation of darker skinned populations in these datasets. We also find no measurable correlation between performance of machine learning model and ITA values, though more comprehensive data is needed for further validation.

Motivation & Objective

• Quantify skin-tone representation in two public dermatology datasets using ITA.
• Develop segmentation masks for non-diseased skin to compute ITA per image.
• Evaluate whether skin-tone (ITA) correlates with classification performance in these datasets.
• Highlight dataset biases and provide guidance for more balanced, fairer dermatologist image datasets.

Proposed method

• Train Mask R-CNN to segment skin lesions and extract non-diseased skin regions.
• Compute ITA from non-diseased skin using L and b from CIELab, bin ITA into 8 categories.
• Train DenseNet-201 models on ISIC2018 (10,015 images, 7 classes) and SD-136 (4,467 images, 136 classes) with appropriate augmentations and class-weighted loss.
• Evaluate classification performance across ITA-based skin-tone categories using validation set accuracy and standard errors from 10 random splits.

Experimental results

Research questions

• RQ1 Do ISIC2018 and SD-136 under-represent darker skin-tone ITA values?
• RQ2 Is there a measurable correlation between ITA-based skin tone and classifier performance in these datasets?
• RQ3 How does skin-tone representation affect classification accuracy across different ITA categories?

Key findings

• Both ISIC2018 and SD-136 predominantly have ITA values in lighter categories, indicating under-representation of darker skin tones.
• Segmentation quality is high for ISIC2018 (accuracy 0.956) and SD-136 (accuracy 0.802), enabling ITA estimation.
• There is no measurable correlation between ITA values and classifier performance within the validation sets for the studied datasets.
• ISIC2018 achieved 0.869 overall accuracy (0.94 very_light to 0.83 tan1 across skin tones); SD-136 achieved 0.604 overall accuracy (0.50 to 0.72 across tones).
• Overall, the study found no clear trend in model performance across ITA-based skin-tone categories, though more data is needed for conclusive claims.