[Paper Review] Towards Reconciling Usability and Usefulness of Explainable AI Methodologies

Pradyumna Tambwekar, Matthew Gombolay|arXiv (Cornell University)|Jan 13, 2023

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This study evaluates four XAI modalities—decision trees, text, programs, and attention maps—for explaining self-driving car decisions in a highway simulation. It finds a key inconsistency: while users subjectively preferred text explanations for usability, decision trees objectively enabled better prediction of car behavior, highlighting the need to balance perceived usability with actual usefulness in personalized XAI design.

ABSTRACT

Interactive Artificial Intelligence (AI) agents are becoming increasingly prevalent in society. However, application of such systems without understanding them can be problematic. Black-box AI systems can lead to liability and accountability issues when they produce an incorrect decision. Explainable AI (XAI) seeks to bridge the knowledge gap, between developers and end-users, by offering insights into how an AI algorithm functions. Many modern algorithms focus on making the AI model "transparent", i.e. unveil the inherent functionality of the agent in a simpler format. However, these approaches do not cater to end-users of these systems, as users may not possess the requisite knowledge to understand these explanations in a reasonable amount of time. Therefore, to be able to develop suitable XAI methods, we need to understand the factors which influence subjective perception and objective usability. In this paper, we present a novel user-study which studies four differing XAI modalities commonly employed in prior work for explaining AI behavior, i.e. Decision Trees, Text, Programs. We study these XAI modalities in the context of explaining the actions of a self-driving car on a highway, as driving is an easily understandable real-world task and self-driving cars is a keen area of interest within the AI community. Our findings highlight internal consistency issues wherein participants perceived language explanations to be significantly more usable, however participants were better able to objectively understand the decision making process of the car through a decision tree explanation. Our work also provides further evidence of importance of integrating user-specific and situational criteria into the design of XAI systems. Our findings show that factors such as computer science experience, and watching the car succeed or fail can impact the perception and usefulness of the explanation.

Motivation & Objective

• To investigate how different XAI modalities affect users' perception and objective understanding of AI decisions in autonomous systems.
• To examine the influence of user-specific factors—such as computer science experience—and situational context (e.g., observing car success or failure) on XAI preference and effectiveness.
• To identify the disconnect between perceived usability and actual usefulness in XAI explanations.
• To inform the design of personalized XAI systems that adapt to individual users’ cognitive and contextual profiles.

Proposed method

• Conducted a large-scale user study with 231 participants using a simulated self-driving car environment to evaluate four XAI modalities: decision trees, natural language text, executable programs, and attention maps.
• Employed a forward simulation protocol where participants predicted car actions after viewing explanations, enabling objective measurement of understanding.
• Collected subjective usability ratings via surveys to assess perceived helpfulness and trust in each modality.
• Analyzed the impact of demographic factors (e.g., computer science background) and situational factors (e.g., watching the car succeed or crash) on explanation perception and performance.
• Used mixed-methods analysis combining quantitative prediction accuracy with qualitative feedback to assess explanation quality.
• Proposed a framework for future personalized XAI systems that adapt explanations based on user disposition and context.

Experimental results

Research questions

• RQ1 How do different XAI modalities (decision trees, text, programs, attention maps) compare in terms of perceived usability and objective understanding of AI decisions?
• RQ2 How do individual factors such as computer science experience influence users’ perception and performance with XAI explanations?
• RQ3 How does situational context—specifically observing the self-driving car succeed or fail—affect users’ trust and evaluation of XAI explanations?
• RQ4 To what extent is there a disconnect between perceived usability and actual usefulness of XAI explanations?
• RQ5 Can a personalized XAI system be designed to dynamically select the most suitable explanation modality based on user characteristics and context?

Key findings

• Text-based explanations were perceived as significantly more usable than other modalities, with participants rating them higher in subjective usability surveys.
• Despite higher perceived usability, text explanations led to lower objective accuracy in predicting the car’s behavior compared to decision trees.
• Decision trees enabled the highest objective understanding, with participants achieving significantly better prediction accuracy than with text or program-based explanations.
• Participants with higher computer science experience showed reduced preference for text-based explanations, suggesting prior knowledge may alter modality preference.
• Observing the car fail (crash) led to a significant decline in users’ attitude toward the XAI agent and reduced trust, especially for text-based explanations.
• The study revealed a fundamental inconsistency: perceived usability (subjective) and actual usefulness (objective) do not align, with text being preferred but less effective for accurate prediction.