[Paper Review] What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use

Sana Tonekaboni, Shalmali Joshi+2 more|arXiv (Cornell University)|May 13, 2019

Explainable Artificial Intelligence (XAI)76 references239 citations

TL;DR

The paper surveys ICU and ED clinicians to define concrete explainability classes and evaluation metrics for translating explainable ML into clinical practice.

ABSTRACT

Translating machine learning (ML) models effectively to clinical practice requires establishing clinicians' trust. Explainability, or the ability of an ML model to justify its outcomes and assist clinicians in rationalizing the model prediction, has been generally understood to be critical to establishing trust. However, the field suffers from the lack of concrete definitions for usable explanations in different settings. To identify specific aspects of explainability that may catalyze building trust in ML models, we surveyed clinicians from two distinct acute care specialties (Intenstive Care Unit and Emergency Department). We use their feedback to characterize when explainability helps to improve clinicians' trust in ML models. We further identify the classes of explanations that clinicians identified as most relevant and crucial for effective translation to clinical practice. Finally, we discern concrete metrics for rigorous evaluation of clinical explainability methods. By integrating perceptions of explainability between clinicians and ML researchers we hope to facilitate the endorsement and broader adoption and sustained use of ML systems in healthcare.

Motivation & Objective

• Identify how explainability can build trust in ML models for clinical practice.
• Characterize the classes of explanations most relevant to clinicians in ICU and ED settings.
• Describe how explainability considerations can be integrated into ML design for reliability and adoption.
• Propose metrics for rigorous evaluation of clinical explainability methods.
• Bridge clinician needs with machine learning research to facilitate translation into practice.

Proposed method

• Upstream stakeholder engagement with clinicians prior to model implementation.
• Qualitative interviews with 10 clinicians (ICU and ED) to explore explainability expectations.
• Use of hypothetical interactive scenarios (CA prediction in ICU; acuity ranking in ED) to probe explanations.
• Interviews conducted until thematic saturation, guided by an interview protocol.
• Identification of explanation classes and their applicability to different clinical settings (ICU vs ED).
• Proposal of evaluation metrics grounded in clinical workflow and decision-making.

Experimental results

Research questions

• RQ1 What aspects of ML explainability help clinicians trust model predictions in real-world practice?
• RQ2 Which classes of explanations are most relevant and actionable for ICU and ED clinicians?
• RQ3 How can explainability be operationalized and evaluated to support clinical translation of ML tools?
• RQ4 What metrics best assess the usefulness and reliability of clinical explainability methods?

Key findings

• Clinicians view explainability as a means to justify predictions within the context of existing medical evidence and practice.
• Feature importance at both patient- and population-level scales is crucial for trust and alignment with clinical judgment.
• Instance-level explanations (e.g., showing similar cases) are context-dependent and less useful in time-constrained ICU/ED settings.
• Uncertainty and confidence scores are valued as part of explanations to manage alarm fatigue and align expectations with clinical actions.
• Temporal explanations and transparent design (e.g., showing decision processes) are important, but care is needed to avoid cognitive overload in high-pressure environments.
• The study provides concrete explanation classes and evaluation metrics to inform future clinical ML research and deployment.