[Paper Review] Skin Lesion Diagnosis using Ensembles, Unscaled Multi-Crop Evaluation and Loss Weighting

Nils Gessert, Thilo Sentker+6 more|arXiv (Cornell University)|Aug 5, 2018

Cutaneous Melanoma Detection and ManagementMedicine16 references65 citations

TL;DR

The paper builds an ensemble of pretrained CNNs with unscaled multi-crop evaluation and loss weighting to address class imbalance for ISIC 2018 skin lesion diagnosis, achieving high weighted accuracy.

ABSTRACT

In this paper we present the methods of our submission to the ISIC 2018 challenge for skin lesion diagnosis (Task 3). The dataset consists of 10000 images with seven image-level classes to be distinguished by an automated algorithm. We employ an ensemble of convolutional neural networks for this task. In particular, we fine-tune pretrained state-of-the-art deep learning models such as Densenet, SENet and ResNeXt. We identify heavy class imbalance as a key problem for this challenge and consider multiple balancing approaches such as loss weighting and balanced batch sampling. Another important feature of our pipeline is the use of a vast amount of unscaled crops for evaluation. Last, we consider meta learning approaches for the final predictions. Our team placed second at the challenge while being the best approach using only publicly available data.

Motivation & Objective

• Address heavy class imbalance in skin lesion classification (HAM10000 and ISIC datasets).
• Develop an ensemble of pretrained CNNs for seven-class lesion diagnosis.
• Evaluate unscaled multi-crop evaluation and loss weighting to improve performance.
• Incorporate meta learning to refine final predictions from multiple models.

Proposed method

• Fine-tune pretrained CNNs (Densenet, SENet, ResNeXt, etc.) on HAM and ISIC data.
• Experiment with loss weighting based on inverse class frequency to balance classes.
• Perform unscaled 224x224 crops and 36-crop evaluation per image for prediction aggregation.
• Use meta-learning (SVM with RBF kernel) on the 36-crop predictions for CV models.
• Combine predictions from CV models and fully trained models by averaging (and compare voting).
• Select final ensemble from top architectures based on 5-fold CV performance.

Experimental results

Research questions

• RQ1 How does heavy class imbalance affect seven-class skin lesion diagnosis performance?
• RQ2 Can an ensemble of state-of-the-art CNNs with unscaled multi-crop evaluation improve weighted accuracy (WACC) on ISIC 2018 tasks?
• RQ3 What is the impact of loss weighting, balanced batching, and meta-learning on final predictions?
• RQ4 Does combining CV-models with fully trained models via averaging yield better results than other fusion methods?

Key findings

• The ensemble achieved the best WACC among tested approaches, with final ensemble performance at 0.851 (WACC).
• SENet154 alone performed best among single models for WACC (0.817 in one setting).
• ISIC data added to HAM training improved some models (e.g., DenseNet121 with ISIC reached WACC 0.804).
• 36-crop evaluation coupled with meta-learning provided gains over single-crop or non-meta approaches for several architectures.
• More recent architectures (Densenet, ResNeXt, PolyNet, SENets) outperformed baselines like Inception/ResNet variants.
• The final ensemble consisted of 54 models and yielded the best overall performance.