[Paper Review] Data, Depth, and Design: Learning Reliable Models for Skin Lesion Analysis

Eduardo Valle, Michel Fornaciali+5 more|arXiv (Cornell University)|Nov 1, 2017

Cutaneous Melanoma Detection and Management4 citations

TL;DR

This paper systematically evaluates nine key hyperparameters in deep learning for skin lesion analysis through exhaustive full-factorial experiments across five test datasets, revealing that data quantity explains nearly half the performance variation. The study demonstrates that transfer learning is critical (accounting for over 60% of performance variation), ensembles outperform sequential optimization, and indirect use of test data for model tuning leads to overoptimistic results—offering a rigorous, reproducible framework for reliable model design in dermatology AI.

ABSTRACT

Deep learning fostered a leap ahead in automated skin lesion analysis in the last two years. Those models are expensive to train and difficult to parameterize. Objective: We investigate methodological issues for designing and evaluating deep learning models for skin lesion analysis. We explore 10 choices faced by researchers: use of transfer learning, model architecture, train dataset, image resolution, type of data augmentation, input normalization, use of segmentation, duration of training, additional use of SVMs, and test data augmentation. Methods: We perform two full factorial experiments, for five different test datasets, resulting in 2560 exhaustive trials in our main experiment, and 1280 trials in our assessment of transfer learning. We analyze both with multi-way ANOVA. We use the exhaustive trials to simulate sequential decisions and ensembles, with and without the use of privileged information from the test set. Results -- main experiment: Amount of train data has disproportionate influence, explaining almost half the variation in performance. Of the other factors, test data augmentation and input resolution are the most influential. Deeper models, when combined, with extra data, also help. -- transfer experiment: Transfer learning is critical, its absence brings huge performance penalties. -- simulations: Ensembles of models are the best option to provide reliable results with limited resources, without using privileged information and sacrificing methodological rigor. Conclusions and Significance: Advancing research on automated skin lesion analysis requires curating larger public datasets. Indirect use of privileged information from the test set to design the models is a subtle, but frequent methodological mistake that leads to overoptimistic results. Ensembles of models are a cost-effective alternative to the expensive full-factorial and to the unstable sequential designs.

Motivation & Objective

• To identify the most influential hyperparameters in deep learning for skin lesion classification through a comprehensive, statistically rigorous evaluation.
• To address the methodological flaw of using test set information during model design, which leads to overoptimistic performance estimates.
• To evaluate the effectiveness of alternative model design strategies—specifically sequential optimization and model ensembling—against the gold-standard full-factorial approach.
• To provide a reproducible, open-source framework for future research in automated skin lesion analysis, emphasizing methodological rigor over performance hype.
• To advocate for larger, publicly curated datasets to advance the field beyond current performance plateaus driven by limited data and biased hyperparameter tuning.

Proposed method

• Conduct two full-factorial experiments: one with nine hyperparameters (e.g., architecture, data augmentation, normalization) across five test datasets (2560 total trials), and another with eight factors focused on transfer learning (1280 trials).
• Use multi-way analysis of variance (ANOVA) to quantify the main effects and interactions of each hyperparameter on model performance (AUC), enabling generalization across datasets.
• Simulate sequential hyperparameter optimization (one factor at a time) and ensemble-based model selection to compare with the exhaustive full-factorial design.
• Implement test data augmentation and privileged information simulation to assess the impact of data leakage during model development.
• Use a standardized pipeline for data acquisition, preprocessing, training, and evaluation, with all code and procedures made publicly available on GitHub.
• Evaluate models on the ISIC 2017 Challenge Part 3 benchmark, achieving state-of-the-art AUCs across all three classification tasks while adhering to methodological constraints.

Experimental results

Research questions

• RQ1 Which hyperparameters have the most significant impact on skin lesion classification performance in deep learning models?
• RQ2 To what extent does transfer learning influence model performance, and how does its absence affect results?
• RQ3 How do different model design strategies—full-factorial, sequential optimization, and ensembling—compare in terms of reliability and performance?
• RQ4 What is the effect of test data augmentation and indirect use of test set information on performance estimation?
• RQ5 Can simple ensembles of models achieve high performance without relying on privileged information or exhaustive hyperparameter search?

Key findings

• The amount of training data explains nearly 48% of the variation in model performance, underscoring the 'unreasonable power of data' in deep learning for skin lesion analysis.
• Transfer learning has a massive impact, accounting for over 60% of the relative variation in performance, and its absence results in significant performance penalties.
• Test data augmentation and input resolution are among the most influential factors after data quantity, with higher resolution and augmented test inputs improving AUCs.
• Deeper models combined with larger training sets yield better performance, indicating that architectural depth and data scale are synergistic.
• The use of segmentation for classification and an additional SVM layer both led to negative performance outcomes, contradicting some prior literature and warranting further investigation.
• Ensembles of models—especially those formed by randomly sampling diverse configurations—achieved state-of-the-art AUCs across all ISIC 2017 Challenge tasks, outperforming both sequential optimization and single-model approaches while maintaining methodological rigor.