[Paper Review] Crowdsourcing Dermatology Images with Google Search Ads: Creating a Real-World Skin Condition Dataset

Abbi Ward, Jimmy Li+18 more|arXiv (Cornell University)|Feb 28, 2024

Cutaneous Melanoma Detection and ManagementMedicine3 citations

TL;DR

This paper presents a scalable crowdsourcing method using Google Search ads to collect real-world dermatology images from the public, resulting in the SCIN dataset of 10,408 images from 5,033 contributors. The dataset includes dermatologist-verified labels, estimated Fitzpatrick Skin Type (eFST), and Monk Skin Tone (eMST), offering broad representation of common, short-duration skin conditions and diverse skin tones, with 97.5% of images being genuine clinical cases.

ABSTRACT

Background: Health datasets from clinical sources do not reflect the breadth and diversity of disease in the real world, impacting research, medical education, and artificial intelligence (AI) tool development. Dermatology is a suitable area to develop and test a new and scalable method to create representative health datasets. Methods: We used Google Search advertisements to invite contributions to an open access dataset of images of dermatology conditions, demographic and symptom information. With informed contributor consent, we describe and release this dataset containing 10,408 images from 5,033 contributions from internet users in the United States over 8 months starting March 2023. The dataset includes dermatologist condition labels as well as estimated Fitzpatrick Skin Type (eFST) and Monk Skin Tone (eMST) labels for the images. Results: We received a median of 22 submissions/day (IQR 14-30). Female (66.72%) and younger (52% < age 40) contributors had a higher representation in the dataset compared to the US population, and 32.6% of contributors reported a non-White racial or ethnic identity. Over 97.5% of contributions were genuine images of skin conditions. Dermatologist confidence in assigning a differential diagnosis increased with the number of available variables, and showed a weaker correlation with image sharpness (Spearman's P values <0.001 and 0.01 respectively). Most contributions were short-duration (54% with onset < 7 days ago ) and 89% were allergic, infectious, or inflammatory conditions. eFST and eMST distributions reflected the geographical origin of the dataset. The dataset is available at github.com/google-research-datasets/scin . Conclusion: Search ads are effective at crowdsourcing images of health conditions. The SCIN dataset bridges important gaps in the availability of representative images of common skin conditions.

Motivation & Objective

• Address the lack of representative, diverse, and real-world dermatology datasets that reflect common, short-duration, and non-malignant skin conditions.
• Overcome biases in clinical datasets that underrepresent darker skin tones and non-White populations.
• Develop a scalable, low-cost method to collect patient-generated dermatology images without targeted demographic sampling.
• Create a publicly available, open-access dataset with clinical-grade labels and skin tone estimates to support fair AI development and medical education.
• Demonstrate the feasibility of using web search ads to recruit individuals actively seeking health information for health data collection.

Proposed method

• Deployed targeted Google Search advertisements using keywords related to common skin symptoms (e.g., rashes, acne, infections) to reach individuals actively searching for dermatological information.
• Directed users to a consented, web-based form to submit self-captured images, along with self-reported symptoms, duration of condition, and demographic information.
• Collected and labeled images with dermatologist-confirmed diagnoses, estimated Fitzpatrick Skin Type (eFST), and Monk Skin Tone (eMST) for each image.
• Used informed consent procedures to ensure ethical data collection and privacy preservation, with data hosted on GitHub for public access.
• Employed statistical analysis to assess label quality, correlation with image quality, and distribution of skin tone and condition types.
• Benchmarked eFST and eMST distributions against known population estimates to assess representativeness of the dataset.

Figure 1: A: Funnel from ad view to submission to the SCIN dataset. B, C: Monthly ad impressions, click-through rates, and submission rates by gender and age

Experimental results

Research questions

• RQ1 Can Google Search ads effectively recruit a diverse, real-world population to contribute dermatology images without demographic targeting?
• RQ2 To what extent does the SCIN dataset reflect the true distribution of common skin conditions, including allergic, infectious, and inflammatory diseases?
• RQ3 How do estimated skin tone (eFST and eMST) distributions in the SCIN dataset compare to known population distributions in the U.S.?
• RQ4 Does the inclusion of self-reported symptoms and image quality improve dermatologist diagnostic confidence in image-based differential diagnosis?
• RQ5 Can crowdsourced, patient-generated images support fairer and more generalizable AI model training compared to clinical-source datasets?

Key findings

• The SCIN dataset contains 10,408 images from 5,033 contributors over 8 months, with a median of 22 submissions per day (IQR 14–30).
• 66.72% of contributors were female, 52% were under age 40, and 32.6% reported a non-White racial or ethnic identity, indicating higher representation of younger and female users compared to the U.S. population.
• Over 97.5% of image contributions were verified as genuine dermatological conditions, with 89% classified as allergic, infectious, or inflammatory.
• Dermatologist diagnostic confidence increased with the number of available variables (symptoms, image quality), and showed a statistically significant weak correlation with image sharpness (Spearman’s ρ < 0.001 and 0.01, respectively).
• The eFST and eMST distributions in the dataset reflected the geographical origin of contributors, with relatively fewer contributors having MST 8+, indicating underrepresentation of the highest skin tone categories.
• The dataset provides a benchmark for FST and MST distributions in the U.S. and enables future research on skin tone and skin type in dermatology, including fairness evaluation of AI models.

Figure 2: A: Sankey diagram illustrating dropout at each step of the data curation process for the SCIN dataset B: Distribution of image sharpness and exposure quality scores C: Dermatologist label confidence by image exposure quality and sharpness score D: Dermatologist label confidence by number o