[Paper Review] Towards Reliable Dermatology Evaluation Benchmarks

Fabian Gröger, Simone Lionetti+7 more|arXiv (Cornell University)|Sep 13, 2023

Cutaneous Melanoma Detection and Management4 citations

TL;DR

This paper proposes a resource-efficient data-cleaning protocol that combines algorithmic ranking with expert confirmation to identify and remove irrelevant samples, near duplicates, and label errors from six public dermatology benchmark datasets. The method reduces performance discrepancies by up to 3.6% in AUPRG, yielding more reliable model evaluation and revised file lists for improved clinical AI benchmarking.

ABSTRACT

Benchmark datasets for digital dermatology unwittingly contain inaccuracies that reduce trust in model performance estimates. We propose a resource-efficient data-cleaning protocol to identify issues that escaped previous curation. The protocol leverages an existing algorithmic cleaning strategy and is followed by a confirmation process terminated by an intuitive stopping criterion. Based on confirmation by multiple dermatologists, we remove irrelevant samples and near duplicates and estimate the percentage of label errors in six dermatology image datasets for model evaluation promoted by the International Skin Imaging Collaboration. Along with this paper, we publish revised file lists for each dataset which should be used for model evaluation. Our work paves the way for more trustworthy performance assessment in digital dermatology.

Motivation & Objective

• To address the lack of reliable evaluation benchmarks in digital dermatology due to data quality issues such as label errors, irrelevant samples, and near duplicates.
• To develop a scalable, resource-efficient protocol that combines algorithmic ranking with expert validation for data curation.
• To reduce performance discrepancies in model evaluation by cleaning highly curated dermatology datasets that still contain hidden inaccuracies.
• To demonstrate that non-expert annotators can reliably identify near-duplicate images, lowering curation costs.
• To release revised, cleaned file lists for six major dermatology datasets to enhance reproducibility and trust in model performance.

Proposed method

• Leverages SelfClean, a method that reformulates data cleaning as ranking problems rather than classification, to identify candidate issues including irrelevant samples, near duplicates, and label errors.
• Applies a confidence-based ranking strategy to prioritize samples most likely to contain data quality issues, reducing the burden on human experts.
• Employs a multi-expert confirmation process where three dermatologists validate ranked candidates, ensuring high reliability in issue detection.
• Uses an intuitive stopping criterion based on expert agreement to terminate the confirmation process efficiently, avoiding exhaustive annotation.
• Incorporates non-expert labeling for near-duplicate detection to assess cost-effectiveness, with statistical validation of agreement levels.
• Releases revised file lists for six benchmark datasets, replacing original evaluation splits to ensure consistent, reliable model evaluation.

Experimental results

Research questions

• RQ1 Can a hybrid protocol combining algorithmic ranking with expert confirmation effectively detect and remove data quality issues in dermatology benchmarks?
• RQ2 To what extent do label errors, irrelevant samples, and near duplicates affect model performance estimates in already-curated dermatology datasets?
• RQ3 Can non-experts reliably identify near-duplicate images in dermatology datasets, reducing the cost of data curation?
• RQ4 How much does performance evaluation vary when using cleaned versus original benchmark datasets?
• RQ5 What is the impact of data quality issues on key metrics like AUROC, AUPRG, and AP in dermatological model evaluation?

Key findings

• The protocol removed up to 1.8% of irrelevant samples and up to 2.8% of near duplicates across the six datasets, significantly reducing noise in evaluation sets.
• Label error prevalence was estimated across all datasets, with the most significant performance degradation observed in AUPRG, where model performance dropped by up to 3.6% on cleaned benchmarks.
• Performance differences of -1.0% in AUROC, -1.4% in AP, and -3.6% in AUPRG were observed when models were evaluated on cleaned versions of the datasets compared to original versions.
• Expert agreement on near-duplicate detection was high and not significantly different from that of non-experts, suggesting that near-duplicate labeling may not require medical expertise.
• The protocol achieved a significant speed-up in curation while maintaining high reliability, with the stopping criterion effectively limiting unnecessary annotation.
• Revised file lists are publicly released at https://github.com/Digital-Dermatology/SelfClean-Revised-Benchmarks to support reproducible and trustworthy model evaluation in digital dermatology.