[Paper Review] Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes

Qian Yang, Aaron Steinfeld+1 more|arXiv (Cornell University)|Apr 21, 2019

Artificial Intelligence in Healthcare and Education24 references69 citations

TL;DR

The paper designs and field-tests a prognostic decision-support tool that silently integrates into clinicians’ decision meetings by generating slides with embedded machine prognostics, aiming for unobtrusive, workflow-friendly AI in critical care. It reports on field evaluation at three VAD centers and interviews with physicians across domains to assess adoption and generalizability.

ABSTRACT

Clinical decision support tools (DST) promise improved healthcare outcomes by offering data-driven insights. While effective in lab settings, almost all DSTs have failed in practice. Empirical research diagnosed poor contextual fit as the cause. This paper describes the design and field evaluation of a radically new form of DST. It automatically generates slides for clinicians' decision meetings with subtly embedded machine prognostics. This design took inspiration from the notion of "Unremarkable Computing", that by augmenting the users' routines technology/AI can have significant importance for the users yet remain unobtrusive. Our field evaluation suggests clinicians are more likely to encounter and embrace such a DST. Drawing on their responses, we discuss the importance and intricacies of finding the right level of unremarkableness in DST design, and share lessons learned in prototyping critical AI systems as a situated experience.

Motivation & Objective

• Investigate why prognostic decision-support tools struggle to be adopted in clinical practice.
• Design a DST that integrates seamlessly into clinicians’ existing decision-making workflow.
• Evaluate whether subtly embedded prognostic information on decision slides is acceptable and effective.
• Assess the generalizability of the unremarkable DST design to other critical medical decisions.

Proposed method

• Embed DST predictions into the top-right corner of automatically generated decision-meeting slides populated from EMR data.
• Use a field-driven design process grounded in Unremarkable Computing to keep the tool subservient to routine workflows.
• Conduct multi-site field studies in three VAD implant centers with one-on-one interviews and observed decision meetings.
• Employ synthetic patient cases for prototyping and evaluate clinicians’ reactions and discussions.
• Analyze data via affinity diagramming and thematic analysis to extract insights about acceptance, practicality, and generalizability.

Experimental results

Research questions

• RQ1 Can a DST be encountered naturally within clinicians’ current decision-making workflow?
• RQ2 Will clinicians accept computational decision support when it is publicly visible during meetings?
• RQ3 Does placing the prediction in a corner provide the right level of unremarkableness to slow decisions only when there is disagreement?
• RQ4 Is the unremarkable DST design generalizable to other critical medical decision meetings?

Key findings

• Clinicians are likely to encounter the DST output when embedded in decision meeting slides across sites.
• There is broad acceptance of prognostic DSTs in the meeting context, with perceived value in providing additional context and a different perspective.
• The right level of unremarkableness is nuanced; the DST slow-down effect is not guaranteed and depends on meeting dynamics and data realism.
• Clinicians prefer models that are validated, locally applicable, and linked to credible evidence; synthetic data pose interpretation challenges.
• Mid-level clinicians feel empowered by clear, visual meeting slides, while senior clinicians maintain control over decision agendas; hierarchy shapes DST use and influence.
• The design shows potential generalizability to other disciplines that use interdisciplinary decision meetings, though real-patient validation and trust are critical for adoption.