[Paper Review] Invisible Users: Uncovering End-Users' Requirements for Explainable AI via Explanation Forms and Goals

Weina Jin, Jianyu Fan+3 more|arXiv (Cornell University)|Feb 10, 2023

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper introduces end-user-friendly explanation forms and explanation goals as a framework to uncover non-technical end-users' requirements for explainable AI (XAI). Through a mixed-method user study with 32 lay participants across four high-stakes tasks, it identifies specific user needs for feature-, example-, and rule-based explanations and 10 key explanation goals—such as trust calibration and bias detection—demonstrating how these insights can directly inform the design and evaluation of user-centered XAI techniques.

ABSTRACT

Non-technical end-users are silent and invisible users of the state-of-the-art explainable artificial intelligence (XAI) technologies. Their demands and requirements for AI explainability are not incorporated into the design and evaluation of XAI techniques, which are developed to explain the rationales of AI decisions to end-users and assist their critical decisions. This makes XAI techniques ineffective or even harmful in high-stakes applications, such as healthcare, criminal justice, finance, and autonomous driving systems. To systematically understand end-users' requirements to support the technical development of XAI, we conducted the EUCA user study with 32 layperson participants in four AI-assisted critical tasks. The study identified comprehensive user requirements for feature-, example-, and rule-based XAI techniques (manifested by the end-user-friendly explanation forms) and XAI evaluation objectives (manifested by the explanation goals), which were shown to be helpful to directly inspire the proposal of new XAI algorithms and evaluation metrics. The EUCA study findings, the identified explanation forms and goals for technical specification, and the EUCA study dataset support the design and evaluation of end-user-centered XAI techniques for accessible, safe, and accountable AI.

Motivation & Objective

• To address the gap in understanding non-technical end-users' requirements for explainable AI (XAI), who are often overlooked in XAI development.
• To overcome the technical communication barrier between XAI systems and end-users by introducing end-user-friendly explanation forms as a bridge.
• To identify and categorize the diverse explanation goals that drive end-users to seek AI explanations in high-stakes decision-making contexts.
• To provide actionable insights for XAI developers by linking end-user requirements to technical specifications and evaluation metrics.
• To support the creation of accessible, safe, and accountable AI systems through user-centered design grounded in empirical user study data.

Proposed method

• Proposed 'end-user-friendly explanation forms' as abstract, user-facing representations of XAI techniques, categorized into feature-, example-, and rule-based forms.
• Designed a mixed-method user study (EUCA) with 32 lay participants performing four AI-assisted critical tasks: medical diagnosis, criminal justice, loan approval, and autonomous driving.
• Collected qualitative feedback on explanation forms and goals using scenario-based tasks, focusing on usability, trust, and decision support.
• Identified 10 core explanation goals driving end-user demand for explanations, including trust calibration, bias detection, and stakeholder communication.
• Used the study findings to generate technical specifications for new XAI algorithms and evaluation metrics aligned with end-user needs.
• Released the EUCA study dataset and materials to promote reproducibility and support future research in user-centered XAI.

Experimental results

Research questions

• RQ1 What types of explanation forms (feature-, example-, rule-based) do non-technical end-users find most useful and understandable in high-stakes AI-assisted tasks?
• RQ2 What are the primary explanation goals that motivate end-users to request AI explanations, and how do these goals vary across different application domains?
• RQ3 How do end-users compare and contrast different explanation forms when making decisions, and what factors influence their preference?
• RQ4 To what extent do existing XAI techniques meet the actual needs of end-users, and in what ways do they fall short?
• RQ5 How can end-user requirements for XAI be systematically translated into technical specifications for new XAI algorithms and evaluation metrics?

Key findings

• End-users consistently preferred example-based explanations (e.g., similar, counterfactual, and typical examples) over feature attribution for clarity and trust calibration in medical and financial decision tasks.
• The most frequently cited explanation goal was 'calibrating trust'—users sought explanations to verify if the AI’s decision aligned with their own judgment or domain knowledge.
• Users expressed strong demand for explanations that help detect model bias, especially in high-stakes domains like criminal justice and healthcare.
• Users struggled with feature-based explanations (e.g., saliency maps) due to lack of interpretability and technical complexity, often leading to confusion or mistrust.
• The study revealed that explanation goals such as 'resolving disagreement between user and AI' and 'improving predicted outcomes' were critical for user engagement and decision confidence.
• The EUCA dataset and findings provide a foundation for developing new XAI evaluation metrics and algorithms tailored to end-user needs, with direct implications for real-world deployment in safety-critical systems.