[论文解读] SkinGPT-4: An Interactive Dermatology Diagnostic System with Visual Large Language Model

Juexiao Zhou, He, Xiaonan+8 more|arXiv (Cornell University)|Apr 21, 2023

Cutaneous Melanoma Detection and Management被引用 16

一句话总结

SkinGPT-4 是一个经过微调的视觉大语言模型，用于皮肤科，可以通过上传图像诊断皮肤疾病，提供解释和治疗建议，并在本地运行以保护隐私。它在150个真实病例上进行了评估，与皮肤科医生对齐的准确性。

ABSTRACT

Skin and subcutaneous diseases rank high among the leading contributors to the global burden of nonfatal diseases, impacting a considerable portion of the population. Nonetheless, the field of dermatology diagnosis faces three significant hurdles. Firstly, there is a shortage of dermatologists accessible to diagnose patients, particularly in rural regions. Secondly, accurately interpreting skin disease images poses a considerable challenge. Lastly, generating patient-friendly diagnostic reports is usually a time-consuming and labor-intensive task for dermatologists. To tackle these challenges, we present SkinGPT-4, which is the world's first interactive dermatology diagnostic system powered by an advanced visual large language model. SkinGPT-4 leverages a fine-tuned version of MiniGPT-4, trained on an extensive collection of skin disease images (comprising 52,929 publicly available and proprietary images) along with clinical concepts and doctors' notes. We designed a two-step training process to allow SkinGPT to express medical features in skin disease images with natural language and make accurate diagnoses of the types of skin diseases. With SkinGPT-4, users could upload their own skin photos for diagnosis, and the system could autonomously evaluate the images, identifies the characteristics and categories of the skin conditions, performs in-depth analysis, and provides interactive treatment recommendations. Meanwhile, SkinGPT-4's local deployment capability and commitment to user privacy also render it an appealing choice for patients in search of a dependable and precise diagnosis of their skin ailments. To demonstrate the robustness of SkinGPT-4, we conducted quantitative evaluations on 150 real-life cases, which were independently reviewed by certified dermatologists, and showed that SkinGPT-4 could provide accurate diagnoses of skin diseases.

研究动机与目标

• 解决皮肤科医生短缺和解读皮肤图像困难的问题，通过实现自诊断与互动式指导。
• 通过在皮肤病图像和临床笔记上对 MiniGPT-4 进行微调，开发面向皮肤科的视觉-语言模型。
• 实现本地部署以提升隐私，同时提供可解释、便于患者理解的诊断报告。
• 评估诊断准确性和实用性，与认证皮肤科医生比较并评估对患者和临床医生的可用性。

提出的方法

• 在一个大型皮肤病图像数据集（52,929 张图像）上对 MiniGPT-4 进行微调，结合临床概念和医生笔记。
• 实施两步训练：步骤1 将视觉特征与自然语言描述的医学概念对齐；步骤2 训练准确的疾病类型诊断。
• 在 LLM 基于诊断之前，使用 Vision Transformer (ViT) 和 Q-Transformer 进行图像编码和嵌入生成。
• 提供一个交互式诊断流程，用户上传图像并获取特征描述、原因和治疗建议。
• 通过本地部署评估隐私性，并将性能与皮肤科医生的评估进行对比。
• 将训练配置为 epochs=20、batch size=2、learning rate=1e-4，在两块 NVIDIA V100 GPU 上进行。

实验结果

研究问题

• RQ1 基于视觉的大语言模型是否能够从用户上传的图像中准确诊断皮肤疾病？
• RQ2 两步式的皮肤科聚焦微调是否比单步方法在诊断准确性和特征描述方面更优？
• RQ3 SkinGPT-4 在真实病例上与认证皮肤科医生的对齐程度及对患者的可用性如何？
• RQ4 本地部署是否在保持诊断有用性的同时解决隐私问题？

主要发现

• 在150个真实病例中，78.76% 的 SkinGPT-4 诊断被认证皮肤科医生评为正确或相关。
• 93.75% 的病例在诊断和解释的总体有用性和相关性上达到一致水平（强烈同意或同意）。
• 医生发现 SkinGPT-4 的原因解释和治疗建议信息丰富且实用（分别为80.63% 和 83.13%）。
• 本地部署的隐私评分较高（91.88%），支持隐私保护的使用场景。
• SkinGPT-4 可以提供 24/7 的初步评估，响应时间快捷（几秒钟），以帮助患者并简化皮肤科医生的工作流程。
• 两步训练在实现准确的形态描述和疾病识别方面优于单步消融。