[Paper Review] Modeling Reliance on XAI Indicating Its Purpose and Attention

Akihiro Maehigashi, Yosuke Fukuchi+1 more|arXiv (Cornell University)|Feb 16, 2023

Ethics and Social Impacts of AI4 citations

TL;DR

This study investigates how explainable AI (XAI) features—specifically, AI purpose disclosure and attention heat maps—affect human trust and reliance in AI. Using structural equation modeling (SEM), it finds that purpose explanations boost trust in easier tasks (compliance), while heat maps only enhance trust when interpretable, especially in difficult tasks (drowsiness detection), where low-interpretability heat maps reduce trust.

ABSTRACT

This study used XAI, which shows its purposes and attention as explanations of its process, and investigated how these explanations affect human trust in and use of AI. In this study, we generated heat maps indicating AI attention, conducted Experiment 1 to confirm the validity of the interpretability of the heat maps, and conducted Experiment 2 to investigate the effects of the purpose and heat maps in terms of reliance (depending on AI) and compliance (accepting answers of AI). The results of structural equation modeling (SEM) analyses showed that (1) displaying the purpose of AI positively and negatively influenced trust depending on the types of AI usage, reliance or compliance, and task difficulty, (2) just displaying the heat maps negatively influenced trust in a more difficult task, and (3) the heat maps positively influenced trust according to their interpretability in a more difficult task.

Motivation & Objective

• To investigate how AI purpose disclosure and attention heat maps influence human trust and reliance in AI systems.
• To examine the role of task difficulty in moderating the effects of XAI explanations on trust and compliance.
• To assess whether interpretability of attention heat maps mediates trust in AI, especially in complex decision-making contexts.
• To address algorithm aversion by evaluating whether transparent AI explanations improve trust calibration and reduce distrust in AI errors.
• To differentiate between reliance (delegating tasks) and compliance (accepting AI outputs) as distinct behavioral outcomes of XAI.

Proposed method

• Generated high- and low-interpretability attention heat maps using Grad-CAM to visualize AI attention in image-based tasks.
• Conducted Experiment 1 to validate the interpretability of heat maps through human perception testing.
• Conducted Experiment 2 with two tasks: an easier obesity screening task and a more difficult drowsiness detection task.
• Collected data on reliance (delegating decisions) and compliance (accepting AI outputs) as behavioral outcomes.
• Applied structural equation modeling (SEM) to analyze causal relationships between purpose disclosure, heat map interpretability, trust (cognitive and emotional), and reliance/compliance.
• Used controlled conditions to isolate the effects of purpose display and attention visualization on trust and behavior.

Experimental results

Research questions

• RQ1 How does displaying the purpose of an AI system affect human trust and reliance in different task contexts?
• RQ2 What is the impact of attention heat maps on trust and compliance, and does this depend on task difficulty?
• RQ3 How does the interpretability of AI-generated heat maps influence cognitive and emotional trust in AI?
• RQ4 Does the presence of heat maps increase or decrease algorithm aversion, particularly in high-difficulty tasks?
• RQ5 How do reliance and compliance differ in their sensitivity to XAI explanations such as purpose and attention?

Key findings

• Displaying the purpose of AI positively influenced trust in the compliance model for the easier obesity screening task, but negatively affected trust in the reliance model for the more difficult drowsiness detection task.
• Simply displaying heat maps without high interpretability negatively influenced trust in the drowsiness detection task, where task difficulty was high.
• Heat map interpretability positively influenced both cognitive and emotional trust in the compliance model for the drowsiness detection task.
• In the drowsiness detection task, higher interpretability of attention heat maps increased emotional trust, suggesting a potential anthropomorphism effect.
• Cognitive trust had a negative influence on reliance and compliance in the drowsiness detection task, possibly due to participants expecting AI to improve through learning.
• The results support the hypothesis that XAI explanations are most effective when they are both meaningful (purpose) and perceptually interpretable (high-quality heat maps), especially in complex, high-stakes contexts.