[Paper Review] Boosting Dermatoscopic Lesion Segmentation via Diffusion Models with Visual and Textual Prompts

Shiyi Du, Xiaosong Wang+6 more|arXiv (Cornell University)|Oct 4, 2023

Cutaneous Melanoma Detection and Management4 citations

TL;DR

This paper proposes a diffusion model-based framework that leverages visual and textual prompts to generate high-fidelity dermatoscopic images with precise lesion characteristics, significantly improving skin lesion segmentation performance. By integrating lesion type, shape, location, and attributes as prompts, the method achieves a 9% gain in SSIM and over 5% improvement in Dice coefficient over prior GAN-based approaches.

ABSTRACT

Image synthesis approaches, e.g., generative adversarial networks, have been popular as a form of data augmentation in medical image analysis tasks. It is primarily beneficial to overcome the shortage of publicly accessible data and associated quality annotations. However, the current techniques often lack control over the detailed contents in generated images, e.g., the type of disease patterns, the location of lesions, and attributes of the diagnosis. In this work, we adapt the latest advance in the generative model, i.e., the diffusion model, with the added control flow using lesion-specific visual and textual prompts for generating dermatoscopic images. We further demonstrate the advantage of our diffusion model-based framework over the classical generation models in both the image quality and boosting the segmentation performance on skin lesions. It can achieve a 9% increase in the SSIM image quality measure and an over 5% increase in Dice coefficients over the prior arts.

Motivation & Objective

• To address the scarcity of annotated dermatoscopic images, especially for rare lesion types and attributes, by generating diverse, high-quality synthetic lesion images.
• To improve the controllability of image generation in medical imaging by incorporating lesion-specific visual and textual prompts.
• To enhance downstream skin lesion segmentation performance through data augmentation using the proposed generation framework.
• To demonstrate the superiority of diffusion models over GANs in generating realistic, detailed dermatoscopic images with precise lesion attributes.
• To develop an automatic lesion shape and mask generation module for scalable data augmentation.

Proposed method

• Utilizes the Stable Diffusion model as a backbone for image generation, conditioned on both visual (segmentation masks) and textual (lesion type and attributes) prompts.
• Employs ControlNet to guide the diffusion process, enabling precise control over lesion shape, location, and appearance during image generation.
• Introduces an automatic module to generate diverse lesion segmentation masks with varying sizes, shapes, and anatomical locations to enrich training data.
• Trains the model using a public dermatoscopic dataset (ISIC), with lesion type and attribute information from ISIC 2017 and 2019 used as textual prompts.
• Applies the generated images as augmented training data for downstream lesion segmentation, using Dice loss and U-Net-based segmentation networks.
• Performs ablation and comparison studies against Pix2PixHD, a strong GAN-based baseline, under identical training and evaluation protocols.

Experimental results

Research questions

• RQ1 Can diffusion models with visual and textual prompts generate dermatoscopic images with higher fidelity and better control over lesion attributes than GANs?
• RQ2 To what extent does data augmentation using the proposed generation framework improve skin lesion segmentation performance?
• RQ3 How does the integration of lesion-specific prompts (type, shape, location, attributes) affect image quality and downstream task accuracy?
• RQ4 Does the automatic lesion mask generation module effectively increase diversity and coverage of synthetic lesion data?
• RQ5 Is the proposed framework transferable to other medical imaging tasks requiring controlled generation of rare or abnormal cases?

Key findings

• The proposed diffusion model-based framework achieves a 9% relative improvement in SSIM over the Pix2PixHD GAN baseline, indicating superior image quality and detail preservation.
• The method improves Dice coefficient by over 5% compared to prior arts, with a final segmentation DSC of 0.914 when using 5K synthetic images.
• Generated images exhibit significantly better texture and lesion structure fidelity, as visually confirmed by zoomed-in regions in Figure 4.
• The performance gain is consistent across different synthetic data scales (1K, 3K, 5K), with minimal marginal gain beyond 3K samples.
• The framework outperforms the SOTA U-Net model (DSC 0.861) and DCSAU-Net (DSC 0.903) when augmented with synthetic data.
• The use of both visual and textual prompts enables precise control over lesion characteristics, enabling targeted data generation for rare or complex cases.