[Paper Review] Out-of-Distribution Detection for Skin Lesion Images with Deep Isolation Forest

Xuan Li, Yuchen Lu+2 more|arXiv (Cornell University)|Mar 20, 2020

Cutaneous Melanoma Detection and Management18 references4 citations

TL;DR

This paper proposes DeepIF, a non-parametric out-of-distribution (OOD) detection method for skin lesion images that combines pre-trained deep convolutional networks with Isolation Forests on feature representations. It achieves state-of-the-art performance, improving OOD detection AUC by 20% over Gaussian-based baselines on the HAM10000 dataset by better capturing intra-class diversity.

ABSTRACT

In this paper, we study the problem of out-of-distribution detection in skin disease images. Publicly available medical datasets normally have a limited number of lesion classes (e.g. HAM10000 has 8 lesion classes). However, there exists a few thousands of clinically identified diseases. Hence, it is important if lesions not in the training data can be differentiated. Toward this goal, we propose DeepIF, a non-parametric Isolation Forest based approach combined with deep convolutional networks. We conduct comprehensive experiments to compare our DeepIF with three baseline models. Results demonstrate state-of-the-art performance of our proposed approach on the task of detecting abnormal skin lesions.

Motivation & Objective

• To address the challenge of detecting skin lesions not present in limited training datasets, such as the 8-class HAM10000 dataset, which underrepresents the thousands of known skin disease types.
• To overcome the limitation of class-conditional Gaussian models in capturing the high intra-class variability of skin lesion images.
• To develop a flexible, non-parametric OOD detection framework that requires only a pre-trained CNN and does not assume a specific distribution for in-distribution or out-of-distribution data.
• To evaluate the effectiveness of feature representations from different convolutional layers for OOD detection in medical imaging.

Proposed method

• The method uses a pre-trained deep CNN to extract features from skin lesion images, leveraging the semantic representations learned during training.
• Isolation Forest (IF) is applied to the extracted features to detect OOD samples, based on the principle that anomalies require fewer splits to isolate in a tree structure.
• Unlike parametric methods, IF does not assume a specific distribution (e.g., Gaussian) for normal data, making it robust to complex, multi-modal distributions in skin lesion features.
• The approach is applied to features from multiple convolutional layers (e.g., f_{-1} to f_{-4}) to analyze the impact of feature depth on OOD detection performance.
• Normality scores are computed using the average path length in the isolation forest, with lower scores indicating higher anomaly likelihood.
• The method is evaluated using standard metrics: AUROC, AUPR, and TNR at 95% TPR across multiple OOD sets (e.g., MEL, BCC, DF, VASC, SCC).

Experimental results

Research questions

• RQ1 Can a non-parametric anomaly detection method like Isolation Forest outperform parametric models such as class-conditional Gaussian in detecting OOD skin lesions with high intra-class variability?
• RQ2 How does the choice of feature layer (from shallow to deep) in a pre-trained CNN affect OOD detection performance?
• RQ3 Does DeepIF achieve better separation between in-distribution and out-of-distribution samples compared to Mahalanobis distance and VAE-based baselines?
• RQ4 Is the performance gain of DeepIF consistent across diverse OOD classes, including rare or visually distinct skin lesions?
• RQ5 Can the last convolutional layer provide the most expressive representations for OOD detection in dermatological imaging?

Key findings

• DeepIF achieves a 20% improvement in OOD detection rate compared to the Gaussian-based Mahalanobis distance method, with a mean AUROC of 0.7136 across all OOD sets.
• The method demonstrates superior separation between in-distribution and OOD samples, as evidenced by clearer score distributions and better ROC curves in Figures 2 and 3.
• Using features from the last convolutional layer (f_{-1}) yields the best OOD detection performance for most lesion types, except for NV, where deeper layers show marginal improvement.
• The performance of DeepIF degrades significantly when using features from shallower layers (f_{-2} to f_{-4}), highlighting the importance of semantic-level representations for anomaly detection.
• For the BCC OOD set, DeepIF achieves an AUROC of 0.7539, significantly outperforming the Mahalanobis baseline (0.5771) and VAE (0.5315).
• The method achieves a TNR of 0.2573 at 95% TPR for the SCC OOD set, indicating strong performance in maintaining specificity under high recall.