[Paper Review] Dermacen Analytica: A Novel Methodology Integrating Multi-Modal Large Language Models with Machine Learning in tele-dermatology

Dimitrios P. Panagoulias, Evridiki Tsoureli‐Nikita+2 more|arXiv (Cornell University)|Mar 21, 2024

Cutaneous Melanoma Detection and ManagementMedicine3 citations

TL;DR

Dermacen Analytica proposes a novel AI-driven workflow integrating multimodal large language models (GPT-4V) with machine learning for tele-dermatology, combining visual and textual analysis to enhance diagnostic accuracy and contextual understanding. The system achieved a weighted score of 0.87 in both diagnostic accuracy and contextual understanding through cross-model validation and expert evaluation.

ABSTRACT

The rise of Artificial Intelligence creates great promise in the field of medical discovery, diagnostics and patient management. However, the vast complexity of all medical domains require a more complex approach that combines machine learning algorithms, classifiers, segmentation algorithms and, lately, large language models. In this paper, we describe, implement and assess an Artificial Intelligence-empowered system and methodology aimed at assisting the diagnosis process of skin lesions and other skin conditions within the field of dermatology that aims to holistically address the diagnostic process in this domain. The workflow integrates large language, transformer-based vision models and sophisticated machine learning tools. This holistic approach achieves a nuanced interpretation of dermatological conditions that simulates and facilitates a dermatologist's workflow. We assess our proposed methodology through a thorough cross-model validation technique embedded in an evaluation pipeline that utilizes publicly available medical case studies of skin conditions and relevant images. To quantitatively score the system performance, advanced machine learning and natural language processing tools are employed which focus on similarity comparison and natural language inference. Additionally, we incorporate a human expert evaluation process based on a structured checklist to further validate our results. We implemented the proposed methodology in a system which achieved approximate (weighted) scores of 0.87 for both contextual understanding and diagnostic accuracy, demonstrating the efficacy of our approach in enhancing dermatological analysis. The proposed methodology is expected to prove useful in the development of next-generation tele-dermatology applications, enhancing remote consultation capabilities and access to care, especially in underserved areas.

Motivation & Objective

• To develop an AI-empowered, holistic diagnostic workflow for skin lesions that mimics dermatologist reasoning.
• To improve diagnostic accuracy and efficiency in tele-dermatology using multimodal AI models.
• To address limitations in remote dermatological diagnosis, especially in underserved regions.
• To integrate explainable AI, segmentation, and evidence-based criteria into a unified diagnostic pipeline.
• To validate the system through cross-model, NLP-based, and expert-annotated evaluation frameworks.

Proposed method

• The system integrates GPT-4V (multimodal LLM) for joint visual and textual understanding of skin lesion images and clinical descriptions.
• It employs advanced machine learning tools for feature extraction, including shape, size, color, and texture analysis of lesions.
• Segmentation algorithms isolate lesions from surrounding skin to enable precise region-of-interest analysis.
• Pragmatic dermatological criteria based on clinical guidelines are embedded to ensure medical relevance and consistency.
• A cross-model validation pipeline uses NLP techniques—similarity comparison and natural language inference (NLI)—to score diagnostic reasoning.
• Human expert evaluation via a structured checklist validates system outputs against gold-standard diagnoses.

Figure 1: Dermacen Analytica, proposed AI-empowered dermatology workflow

Experimental results

Research questions

• RQ1 Can a multimodal LLM-based system achieve high diagnostic accuracy and contextual reasoning in tele-dermatology?
• RQ2 How does the integration of vision transformers and NLP improve diagnostic consistency and explainability?
• RQ3 To what extent does the system’s performance match that of human dermatologists in diagnostic reasoning and accuracy?
• RQ4 Can the system reduce diagnostic errors and hallucinations through multi-model collaboration and validation?
• RQ5 How effective is the system in enhancing access to dermatological care in remote or underserved areas?

Key findings

• The system achieved a weighted score of 0.87 in both diagnostic accuracy and contextual understanding, indicating strong performance.
• NLP-based evaluation using natural language inference (NLI) and similarity scoring confirmed high alignment between correct and predicted diagnoses.
• Human expert evaluation yielded a mean score of 4.31 out of 5 for diagnostic reasoning, equivalent to 0.86 in normalized terms.
• The cross-model validation pipeline effectively reduced hallucinations and improved diagnostic reliability through multi-modal consistency checks.
• The system demonstrated strong adaptability to diverse skin conditions and lesions using evidence-based evaluation criteria.
• The methodology is scalable and suitable for deployment in next-generation tele-dermatology applications, especially in low-resource settings.

Figure 2: Reducing the knowledge space