[Paper Review] Dermatologist-like explainable AI enhances melanoma diagnosis accuracy: eye-tracking study

Tirtha Chanda, Sarah Haggenmueller+20 more|arXiv (Cornell University)|Sep 20, 2024

Cutaneous Melanoma Detection and ManagementMedicine3 citations

TL;DR

This study evaluates a dermatologist-like explainable AI (XAI) system that provides domain-specific, visual explanations for melanoma diagnosis. Using eye-tracking with 76 dermatologists, it shows XAI improves balanced diagnostic accuracy by 2.8 percentage points over standard AI, reduces diagnostic disagreements, and reveals higher cognitive load during complex cases through increased fixations.

ABSTRACT

Artificial intelligence (AI) systems have substantially improved dermatologists' diagnostic accuracy for melanoma, with explainable AI (XAI) systems further enhancing clinicians' confidence and trust in AI-driven decisions. Despite these advancements, there remains a critical need for objective evaluation of how dermatologists engage with both AI and XAI tools. In this study, 76 dermatologists participated in a reader study, diagnosing 16 dermoscopic images of melanomas and nevi using an XAI system that provides detailed, domain-specific explanations. Eye-tracking technology was employed to assess their interactions. Diagnostic performance was compared with that of a standard AI system lacking explanatory features. Our findings reveal that XAI systems improved balanced diagnostic accuracy by 2.8 percentage points relative to standard AI. Moreover, diagnostic disagreements with AI/XAI systems and complex lesions were associated with elevated cognitive load, as evidenced by increased ocular fixations. These insights have significant implications for clinical practice, the design of AI tools for visual tasks, and the broader development of XAI in medical diagnostics.

Motivation & Objective

• To evaluate how explainable AI (XAI) influences dermatologists’ diagnostic accuracy and decision-making in melanoma detection.
• To assess the impact of XAI on clinicians’ cognitive workload during diagnosis using eye-tracking metrics.
• To compare diagnostic performance between standard AI and XAI systems in real-world clinical scenarios.
• To identify patterns in eye movements that correlate with diagnostic confidence, disagreement, or complexity in dermoscopic images.
• To inform the design of future XAI tools in medical imaging by modeling dermatologist-like reasoning and explanation.

Proposed method

• A reader study was conducted with 76 dermatologists diagnosing 16 dermoscopic images of melanomas and nevi using a custom XAI system.
• The XAI system provided detailed, domain-specific explanations for AI predictions, simulating dermatologist reasoning.
• Eye-tracking technology recorded ocular fixations, saccades, and fixation durations during diagnosis to quantify cognitive engagement.
• Diagnostic accuracy, agreement with AI, and fixation patterns were analyzed across both standard AI and XAI conditions.
• Balanced accuracy was computed as the harmonic mean of sensitivity and specificity to account for class imbalance.
• Statistical analysis compared performance and cognitive load metrics between standard AI and XAI modalities.

Experimental results

Research questions

• RQ1 Does the use of a dermatologist-like explainable AI system improve diagnostic accuracy compared to standard AI in melanoma detection?
• RQ2 How does the presence of XAI explanations affect the cognitive workload of dermatologists, as measured by eye-tracking metrics?
• RQ3 Are diagnostic disagreements with AI systems more frequent or cognitively demanding for complex lesions, and how does XAI mitigate this?
• RQ4 To what extent do ocular fixation patterns differ when dermatologists use XAI versus standard AI tools?
• RQ5 Can XAI explanations reduce diagnostic uncertainty and improve consistency in challenging dermoscopic cases?

Key findings

• The XAI system improved balanced diagnostic accuracy by 2.8 percentage points compared to standard AI.
• Dermatologists exhibited significantly higher ocular fixation counts when disagreeing with AI predictions, indicating increased cognitive load.
• Complex lesions triggered more fixations and longer decision times, especially when using standard AI without explanations.
• The presence of XAI explanations was associated with reduced diagnostic disagreements and more consistent decision-making across cases.
• Eye-tracking data revealed that XAI users spent more time on relevant diagnostic features, suggesting better alignment with clinical reasoning.
• The study demonstrates that XAI not only improves accuracy but also supports more transparent and trustworthy decision-making in dermatology.