[Paper Review] SkinGEN: an Explainable Dermatology Diagnosis-to-Generation Framework with Interactive Vision-Language Models

Bo Lin, Yingjing Xu+4 more|arXiv (Cornell University)|Apr 23, 2024

Biomedical Text Mining and Ontologies4 citations

TL;DR

SkinGEN is a novel, interactive vision-language framework that enhances dermatological diagnosis explainability by generating personalized, realistic skin condition images from VLM predictions using Stable Diffusion and LoRA fine-tuning. It significantly improves user trust, comprehension, and perceived usability compared to baseline models, demonstrating superior visual explainability in dermatology applications.

ABSTRACT

With the continuous advancement of vision language models (VLMs) technology, remarkable research achievements have emerged in the dermatology field, the fourth most prevalent human disease category. However, despite these advancements, VLM still faces explainable problems to user in diagnosis due to the inherent complexity of dermatological conditions, existing tools offer relatively limited support for user comprehension. We propose SkinGEN, a diagnosis-to-generation framework that leverages the stable diffusion(SD) model to generate reference demonstrations from diagnosis results provided by VLM, thereby enhancing the visual explainability for users. Through extensive experiments with Low-Rank Adaptation (LoRA), we identify optimal strategies for skin condition image generation. We conduct a user study with 32 participants evaluating both the system performance and explainability. Results demonstrate that SkinGEN significantly improves users' comprehension of VLM predictions and fosters increased trust in the diagnostic process. This work paves the way for more transparent and user-centric VLM applications in dermatology and beyond.

Motivation & Objective

• To address the lack of visual explainability in dermatological vision-language models (VLMs), which limits user understanding and trust.
• To reduce hallucination and cognitive load in VLM-based dermatological diagnosis by providing intuitive, visual demonstrations of diagnosed and alternative skin conditions.
• To develop a user-centric framework that integrates diagnosis with on-demand, personalized image generation for improved clinical interpretability.
• To evaluate the impact of visual explainability on user perception, trust, and usability in dermatological decision support systems.
• To establish a new paradigm for transparent, interactive VLM applications in dermatology and beyond through controlled image generation.

Proposed method

• SkinGEN employs a two-stage pipeline: first, a VLM analyzes a user-uploaded skin image and provides a primary diagnosis with potential differential diagnoses.
• Second, the system generates masked representations of the affected skin area to guide conditional image generation.
• Using Low-Rank Adaptation (LoRA) and IP-Adapter fine-tuning, the framework controls Stable Diffusion to generate high-fidelity, realistic visualizations of the diagnosed and alternative skin conditions.
• The generated images serve as visual explanations, enabling users to compare symptoms and understand diagnostic reasoning.
• User interactions are supported through a chat-based interface that allows clarification of diagnoses and retrieval of similar condition examples.
• The system is evaluated via a user study with 32 participants comparing SkinGEN against two baseline systems in terms of trust, comprehension, and cognitive effort.

Experimental results

Research questions

• RQ1 How does the integration of visual generation with VLM-based dermatological diagnosis affect user trust in the system?
• RQ2 To what extent does generating personalized visual examples improve user comprehension of complex dermatological diagnoses?
• RQ3 How does SkinGEN compare to baseline systems in terms of perceived usability and cognitive effort?
• RQ4 What fine-tuning strategies (e.g., LoRA, IP-Adapter) yield the most realistic and relevant skin disease image generations?
• RQ5 Can interactive, vision-language-based image generation enhance the interpretability of VLM predictions in dermatology?

Key findings

• SkinGEN significantly outperformed baseline systems in perceived trust, with users rating it as more trustworthy (p < 0.05) in the user study.
• Users found SkinGEN’s explanations easier to understand and required less cognitive effort compared to plain VLM outputs or case-based retrieval systems.
• The mean score for perceived realism of generated skin disease images was 4.16 out of 5, indicating high visual fidelity and user acceptance.
• Users reported high utility, with a mean score of 4.38 for overall system usefulness and 4.31 for willingness to use it in the future.
• The integration of LoRA and IP-Adapter enabled stable and controllable image generation, with optimized prompting and adapter configurations yielding the best results.
• The framework effectively reduces diagnostic ambiguity by enabling visual comparison of similar skin conditions, enhancing user confidence and understanding.