[Paper Review] Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosis

Shao Zhang, Jianing Yu+9 more|arXiv (Cornell University)|Sep 17, 2023

Sepsis Diagnosis and Treatment7 citations

TL;DR

The paper analyzes why existing AI Sepsis prediction modules are not trusted in clinical practice and introduces SepsisLab, a human-centered system that supports intermediate decision-making stages and future risk prediction with actionable lab-test suggestions.

ABSTRACT

Today's AI systems for medical decision support often succeed on benchmark datasets in research papers but fail in real-world deployment. This work focuses on the decision making of sepsis, an acute life-threatening systematic infection that requires an early diagnosis with high uncertainty from the clinician. Our aim is to explore the design requirements for AI systems that can support clinical experts in making better decisions for the early diagnosis of sepsis. The study begins with a formative study investigating why clinical experts abandon an existing AI-powered Sepsis predictive module in their electrical health record (EHR) system. We argue that a human-centered AI system needs to support human experts in the intermediate stages of a medical decision-making process (e.g., generating hypotheses or gathering data), instead of focusing only on the final decision. Therefore, we build SepsisLab based on a state-of-the-art AI algorithm and extend it to predict the future projection of sepsis development, visualize the prediction uncertainty, and propose actionable suggestions (i.e., which additional laboratory tests can be collected) to reduce such uncertainty. Through heuristic evaluation with six clinicians using our prototype system, we demonstrate that SepsisLab enables a promising human-AI collaboration paradigm for the future of AI-assisted sepsis diagnosis and other high-stakes medical decision making.

Motivation & Objective

• Understand why clinicians abandon current AI-based sepsis modules in EHR systems.
• Design a human-centered AI assistant that supports intermediate stages of medical decision making in sepsis diagnosis.
• Develop SepsisLab to predict current and future sepsis risk with uncertainty visualization.
• Provide actionable lab-test recommendations to reduce uncertainty and improve final decisions.
• Evaluate whether SepsisLab improves perceived human-AI collaboration among clinicians.

Proposed method

• Conduct an exploratory formative study with six clinicians using the Epic Sepsis Module (ESM) to identify design gaps.
• Design and implement SepsisLab, an AI-assisted system predicting current and future sepsis risk.
• Incorporate uncertainty visualization and counterfactual lab-test recommendations to reduce decision uncertainty.
• Enable ranking of the top lab tests that could reduce uncertainty and provide interactive experimentation through counterfactuals.
• Employ a user-centered interface aligned with clinical workflow to facilitate hypothesis generation, data gathering, and hypothesis testing (intermediate decision stages).
• Ground the prototype with de-identified data from MIMIC-III to demonstrate integration with existing EHR systems.

Figure 1. Existing EPIC Sepsis Module and Our Proposed Sepsis Decision-Support Module in Medical Decision Making Workflow . Our work focuses on sepsis diagnosis, a high-uncertainty, high-stakes, time-sensitive medical decision-making process. Physicians usually take four steps: (1) generating hypoth

Experimental results

Research questions

• RQ1 Why is the current AI sepsis module perceived as useless or intimidating by clinicians?
• RQ2 How should an AI system be designed to support intermediate medical decision-making stages (hypothesis generation, data gathering, hypothesis testing) in sepsis diagnosis?
• RQ3 Can an AI tool predict future sepsis risk with uncertainty visualization and actionable data-gathering recommendations to improve final decisions?

Key findings

• Clinicians find the existing Epic Sepsis Module (ESM) belated, inaccurate, hard to interpret, and lacking actionable insights.
• The current AI paradigm is experienced as a competition rather than collaboration, undermining trust and adoption.
• SepsisLab demonstrates a promising human-AI collaboration by predicting current and near-future sepsis risk, visualizing uncertainty, and recommending lab tests to reduce uncertainty.
• Clinicians favor AI that supports intermediate decisions (hypotheses generation and data gathering) rather than solely final predictions.
• User evaluations indicate clinicians view SepsisLab as a collaborator rather than an intimidating predictor.

Figure 2. Existing Human-AI Interaction and “Competition” Paradigm. The current sepsis module mainly focuses on supporting the final decision-making stage, yet physicians often find the AI predictions are too late and not helpful.