[Paper Review] Transformer-based interpretable multi-modal data fusion for skin lesion classification

Theodor Cheslerean-Boghiu, Melia-Evelina Fleischmann+2 more|arXiv (Cornell University)|Apr 3, 2023

Cutaneous Melanoma Detection and Management4 citations

TL;DR

This paper proposes a transformer-based, interpretable multi-modal fusion model for skin lesion classification that integrates dermoscopic images and patient metadata in a single-stage process using self-attention mechanisms. The method achieves state-of-the-art performance in both image-rich and metadata-rich settings while providing native interpretability through saliency and relevancy maps, enabling clinicians to validate model decisions and build trust in AI-driven diagnostics.

ABSTRACT

A lot of deep learning (DL) research these days is mainly focused on improving quantitative metrics regardless of other factors. In human-centered applications, like skin lesion classification in dermatology, DL-driven clinical decision support systems are still in their infancy due to the limited transparency of their decision-making process. Moreover, the lack of procedures that can explain the behavior of trained DL algorithms leads to almost no trust from clinical physicians. To diagnose skin lesions, dermatologists rely on visual assessment of the disease and the data gathered from the patient's anamnesis. Data-driven algorithms dealing with multi-modal data are limited by the separation of feature-level and decision-level fusion procedures required by convolutional architectures. To address this issue, we enable single-stage multi-modal data fusion via the attention mechanism of transformer-based architectures to aid in diagnosing skin diseases. Our method beats other state-of-the-art single- and multi-modal DL architectures in image-rich and patient-data-rich environments. Additionally, the choice of the architecture enables native interpretability support for the classification task both in the image and metadata domain with no additional modifications necessary.

Motivation & Objective

• Address the lack of transparency and clinical trust in deep learning-based dermatological decision support systems.
• Overcome limitations of traditional fusion methods in convolutional neural networks that separate feature-level and decision-level fusion.
• Enable single-stage, end-to-end multi-modal data fusion using the attention mechanism of transformers for improved performance.
• Provide native interpretability for both image and metadata modalities without additional modifications.
• Ensure clinical relevance by aligning model interpretations with known dermatological diagnostic criteria and patient history factors.

Proposed method

• Employ a transformer-based architecture to perform joint encoding of dermoscopic images and tabular patient metadata in a shared latent space.
• Use multi-head self-attention to dynamically weigh the importance of image patches and metadata features during classification.
• Perform joint embedding of image and tabular data through modality-specific tokenizers and learnable position embeddings.
• Apply a classification head on the [CLS] token to predict skin lesion diagnosis from fused representations.
• Leverage Grad-CAM and gradient-based saliency maps for visual interpretability of image attention.
• Use LIME-style feature attribution to compute relevancy scores for individual metadata factors, enabling clinical interpretation of model decisions.

Experimental results

Research questions

• RQ1 Can a single-stage transformer-based architecture outperform existing state-of-the-art models in multi-modal skin lesion classification?
• RQ2 To what extent can attention mechanisms in transformers enable effective fusion of dermoscopic images and clinical metadata?
• RQ3 Does the model’s attention mechanism highlight clinically relevant features in both images and metadata, such as lesion growth behavior or anatomical location?
• RQ4 Can the model’s interpretability outputs (saliency and relevancy maps) help clinicians validate or correct model predictions?
• RQ5 How does the model perform on rare or visually ambiguous skin lesions with high intra-class variance?

Key findings

• The proposed model achieves state-of-the-art performance in both image-rich and patient-data-rich environments, outperforming existing single- and multi-modal deep learning architectures.
• The model demonstrates superior localization of relevant lesion regions in dermoscopic images, even in low-contrast or hair-occluded cases.
• Saliency maps reveal that the model correctly identifies clinically relevant features such as changing behavior in melanoma and anatomical location in basal cell carcinoma.
• Relevancy maps show that the model learns meaningful associations between metadata and disease: e.g., age for nevi, gender and location for BCC, and growth behavior for SCC.
• The model avoids misclassifying based on non-medical cues (e.g., markers) despite their attention, indicating robustness to spurious correlations.
• For sparse classes like ACK and SEK, localization is less precise due to limited intra-class variation, but the model still generalizes well for high-variance classes like SCC.