[Paper Review] The Utility of Explainable AI in Ad Hoc Human-Machine Teaming

Rohan Paleja, Muyleng Ghuy+3 more|arXiv (Cornell University)|Sep 8, 2022

Human-Automation Interaction and Safety19 citations

TL;DR

This paper investigates the impact of Explainable AI (xAI) on ad hoc human-machine teaming using two human-subject experiments. It finds that xAI enhances situational awareness and performance for novices but degrades expert performance due to cognitive overhead, demonstrating that xAI benefits are highly dependent on team composition and must be tailored to user expertise.

ABSTRACT

Recent advances in machine learning have led to growing interest in Explainable AI (xAI) to enable humans to gain insight into the decision-making of machine learning models. Despite this recent interest, the utility of xAI techniques has not yet been characterized in human-machine teaming. Importantly, xAI offers the promise of enhancing team situational awareness (SA) and shared mental model development, which are the key characteristics of effective human-machine teams. Rapidly developing such mental models is especially critical in ad hoc human-machine teaming, where agents do not have a priori knowledge of others' decision-making strategies. In this paper, we present two novel human-subject experiments quantifying the benefits of deploying xAI techniques within a human-machine teaming scenario. First, we show that xAI techniques can support SA ($p<0.05)$. Second, we examine how different SA levels induced via a collaborative AI policy abstraction affect ad hoc human-machine teaming performance. Importantly, we find that the benefits of xAI are not universal, as there is a strong dependence on the composition of the human-machine team. Novices benefit from xAI providing increased SA ($p<0.05$) but are susceptible to cognitive overhead ($p<0.05$). On the other hand, expert performance degrades with the addition of xAI-based support ($p<0.05$), indicating that the cost of paying attention to the xAI outweighs the benefits obtained from being provided additional information to enhance SA. Our results demonstrate that researchers must deliberately design and deploy the right xAI techniques in the right scenario by carefully considering human-machine team composition and how the xAI method augments SA.

Motivation & Objective

• To quantify the utility of Explainable AI (xAI) in enhancing situational awareness (SA) and shared mental model development in ad hoc human-machine teaming.
• To examine how different xAI techniques—specifically status-based and decision-tree explanations—affect team performance across varying levels of human expertise.
• To investigate whether xAI support universally improves human-machine teaming performance or if its effectiveness depends on human cognitive load and prior experience.
• To identify design principles for xAI in human-machine teams by analyzing the trade-off between information gain and cognitive overhead.

Proposed method

• Conducted two controlled human-subject experiments using a Minecraft-based collaborative task environment to simulate ad hoc human-machine teaming.
• Deployed a cobot with two distinct xAI techniques: a simple status-based explanation (e.g., 'working on task X') and a decision-tree-based explanation of the cobot’s policy.
• Varied the level of xAI support across conditions, including partial (IV1) and complete (IV2) explanation sets, to assess the impact of explanation complexity.
• Measured objective performance (task completion time, success rate), situational awareness (SA), and subjective metrics (trust, perceived capability, working alliance).
• Classified participants into novice and expert groups based on prior experience with the game to analyze expertise-dependent effects.
• Used statistical analysis (p < 0.05) to evaluate significance of performance and perception differences across xAI conditions and expertise levels.

Experimental results

Research questions

• RQ1 Does the deployment of xAI techniques improve situational awareness (SA) in ad hoc human-machine teaming?
• RQ2 How does the composition of the human-machine team—specifically, the expertise level of the human—moderate the effectiveness of xAI support?
• RQ3 Do different types of xAI explanations (status vs. decision-tree) yield varying performance outcomes for novices versus experts?
• RQ4 Does increased explanation complexity (e.g., full decision tree) lead to performance degradation due to cognitive overload, especially among experts?
• RQ5 To what extent does xAI influence subjective perceptions of trust, capability, and team cohesion in human-machine teams?

Key findings

• xAI-based support significantly improves situational awareness (SA) for human teammates, with p < 0.05, confirming that xAI enhances the ability to perform situational analysis.
• Novices benefit from status-based xAI support, achieving better performance (p < 0.05), while decision-tree-based explanations do not improve their performance and may add cognitive burden.
• Expert performance degrades when xAI support is provided (p < 0.05), indicating that the cognitive cost of processing explanations outweighs the benefits of increased SA.
• Experts perceive xAI-augmented cobots as more trustworthy, intelligent, and capable, but this perception does not translate into performance gains.
• Performance degrades further when experts receive complete xAI support (IV2) compared to partial support (IV1), suggesting that explanation complexity exacerbates cognitive overload.
• All participants, regardless of expertise, rated xAI-augmented cobots more highly in terms of trust, perceived capability, and working alliance, indicating a strong positive perception of xAI despite performance trade-offs.