[Paper Review] SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained model debugging and analysis

Roxana Daneshjou, Mert Yüksekgönül+3 more|arXiv (Cornell University)|Feb 1, 2023

Cutaneous Melanoma Detection and Management4 citations

TL;DR

SkinCon is a dermatology dataset of 3,230 images from the Fitzpatrick 17k and 656 from the Diverse Dermatology Images dataset, densely annotated by dermatologists with 48 clinically relevant skin lesion concepts such as 'plaque', 'scale', and 'erosion'. The dataset enables fine-grained model debugging and interpretable AI through concept-based explanations and post-hoc concept bottleneck models, with 22 concepts having at least 50 images and 32 having over 30, making it the first medical dataset with broad, multi-disease concept annotations.

ABSTRACT

For the deployment of artificial intelligence (AI) in high-risk settings, such as healthcare, methods that provide interpretability/explainability or allow fine-grained error analysis are critical. Many recent methods for interpretability/explainability and fine-grained error analysis use concepts, which are meta-labels that are semantically meaningful to humans. However, there are only a few datasets that include concept-level meta-labels and most of these meta-labels are relevant for natural images that do not require domain expertise. Densely annotated datasets in medicine focused on meta-labels that are relevant to a single disease such as melanoma. In dermatology, skin disease is described using an established clinical lexicon that allows clinicians to describe physical exam findings to one another. To provide a medical dataset densely annotated by domain experts with annotations useful across multiple disease processes, we developed SkinCon: a skin disease dataset densely annotated by dermatologists. SkinCon includes 3230 images from the Fitzpatrick 17k dataset densely annotated with 48 clinical concepts, 22 of which have at least 50 images representing the concept. The concepts used were chosen by two dermatologists considering the clinical descriptor terms used to describe skin lesions. Examples include "plaque", "scale", and "erosion". The same concepts were also used to label 656 skin disease images from the Diverse Dermatology Images dataset, providing an additional external dataset with diverse skin tone representations. We review the potential applications for the SkinCon dataset, such as probing models, concept-based explanations, and concept bottlenecks. Furthermore, we use SkinCon to demonstrate two of these use cases: debugging mistakes of an existing dermatology AI model with concepts and developing interpretable models with post-hoc concept bottleneck models.

Motivation & Objective

• To address the lack of medically relevant, multi-disease concept annotations in healthcare AI datasets.
• To enable fine-grained error analysis and model interpretability in dermatology AI by providing domain expert-labeled clinical descriptors.
• To support the development of concept-based explanations and post-hoc interpretable models using a standardized clinical lexicon.
• To improve model robustness and fairness by enabling analysis across diverse skin tones and lesion types.
• To create a foundational dataset for future research in interpretable, generalizable dermatology AI.

Proposed method

• Dermatologists selected 48 clinically meaningful skin lesion descriptors (e.g., 'plaque', 'scale') based on established dermatological terminology.
• Images from the Fitzpatrick 17k and Diverse Dermatology Images (DDI) datasets were labeled with these 48 concepts using a two-stage annotation and validation process.
• The dataset includes 3,230 images from Fitzpatrick 17k and 656 from DDI, with 25 concepts having >50 images and 32 having >30 images.
• Concept-based model interpretation was demonstrated using conceptual counterfactual explanations to debug mispredictions in an existing dermatology AI model.
• Post-hoc concept bottleneck models were trained using SkinCon concepts to improve model interpretability and performance.
• The dataset was released at https://SkinCon-dataset.github.io to support reproducible research in medical AI.

Experimental results

Research questions

• RQ1 Can a densely annotated dermatology dataset with clinically relevant, multi-disease concepts improve model interpretability and error analysis?
• RQ2 How effective are concept-based explanations in identifying and diagnosing model mispredictions in dermatology AI?
• RQ3 To what extent can post-hoc concept bottleneck models trained on SkinCon improve the interpretability of existing dermatology classifiers?
• RQ4 How does the inclusion of diverse skin tones in SkinCon impact the fairness and generalizability of model error analysis?
• RQ5 What are the limitations of current clinical lexicons and annotation practices when applied to AI model development in dermatology?

Key findings

• SkinCon includes 3,230 images from Fitzpatrick 17k and 656 from DDI, with 48 clinical concepts annotated by dermatologists, 22 of which have at least 50 images.
• The dataset enables concept-based explanations that reveal why a model misclassifies a skin lesion, such as attributing a misdiagnosis to the presence of 'scale' or 'erosion'.
• Post-hoc concept bottleneck models trained on SkinCon concepts improved model interpretability and allowed for error analysis based on clinical features.
• The dataset supports slice discovery and probing of model behavior across different lesion types and skin tones.
• Despite efforts, SkinCon remains imbalanced, with common concepts like 'papule' and 'plaque' overrepresented and rarer features like 'nodule' underrepresented.
• The dataset includes a higher proportion of Fitzpatrick skin types I–IV than V–VI, highlighting a need for future expansion to improve representation of darker skin tones.