[Paper Review] Question-Driven Design Process for Explainable AI User Experiences

Q. Vera Liao, Milena Pribic+3 more|arXiv (Cornell University)|Apr 8, 2021

Explainable Artificial Intelligence (XAI)52 references19 citations

TL;DR

This paper proposes a Question-Driven Design Process for Explainable AI (XAI) user experiences, grounding UX design in users' actual questions to bridge the gap between technical XAI techniques and real-world user needs. By mapping prototypical user questions to exemplar XAI techniques, the method enables designers and AI engineers to collaboratively select and implement appropriate explanations, improving alignment between user understanding, technical feasibility, and design goals in AI systems.

ABSTRACT

A pervasive design issue of AI systems is their explainability--how to provide appropriate information to help users understand the AI. The technical field of explainable AI (XAI) has produced a rich toolbox of techniques. Designers are now tasked with the challenges of how to select the most suitable XAI techniques and translate them into UX solutions. Informed by our previous work studying design challenges around XAI UX, this work proposes a design process to tackle these challenges. We review our and related prior work to identify requirements that the process should fulfill, and accordingly, propose a Question-Driven Design Process that grounds the user needs, choices of XAI techniques, design, and evaluation of XAI UX all in the user questions. We provide a mapping guide between prototypical user questions and exemplars of XAI techniques to reframe the technical space of XAI, also serving as boundary objects to support collaboration between designers and AI engineers. We demonstrate it with a use case of designing XAI for healthcare adverse events prediction, and discuss lessons learned for tackling design challenges of AI systems.

Motivation & Objective

• To address the critical design challenge of translating technical XAI techniques into effective, user-centered UX solutions.
• To support designers in selecting appropriate XAI techniques by grounding choices in real user questions rather than technical capabilities.
• To improve collaboration between designers and AI engineers through shared understanding using a mapping guide as a boundary object.
• To provide a structured, iterative design process that aligns XAI UX development with user needs, context, and evaluation criteria.
• To reframe the technical XAI toolbox around human-centered values and user experience goals, moving beyond model-centric explanations.

Proposed method

• The design process is centered on identifying and prioritizing user questions that reflect user needs for understanding AI decisions.
• A mapping guide is developed to link prototypical user questions (e.g., 'Why was this decision made?') to specific XAI techniques (e.g., LIME, SHAP, counterfactuals).
• The method supports iterative design and evaluation by anchoring UX decisions in user questions, ensuring explanations are relevant and actionable.
• Designers and AI engineers use the mapping guide as a shared reference to co-develop solutions, aligning technical affordances with user expectations.
• The process is tested and refined through two real-world use cases, including healthcare adverse event prediction, to validate its practical applicability.
• User research findings are used to define the initial set of user questions, ensuring the design remains grounded in actual user contexts and values.

Experimental results

Research questions

• RQ1 How can designers effectively select appropriate XAI techniques that align with real user needs rather than technical capabilities?
• RQ2 What is the role of user questions in guiding the design of explainable AI user experiences?
• RQ3 How can designers and AI engineers collaborate more effectively when developing XAI UX solutions?
• RQ4 In what ways can XAI techniques be systematically mapped to user questions to support design decisions?
• RQ5 How does grounding XAI UX design in user questions improve the usability and trustworthiness of AI systems?

Key findings

• The Question-Driven Design Process successfully aligns XAI UX design with user needs by anchoring all design decisions in actual user questions.
• The mapping guide between user questions and XAI techniques serves as an effective boundary object, enabling shared understanding between designers and AI engineers.
• The process enables a designerly understanding of XAI techniques by framing their affordances around user comprehension goals rather than technical features.
• The method supports iterative refinement of both design and technical solutions through shared evaluation criteria rooted in user questions.
• The approach was validated through two use cases, demonstrating its practical utility in complex domains such as healthcare AI.
• The study highlights that current XAI toolboxes are often misaligned with end-user needs, and this method helps reframe them around human-centered values.