[Paper Review] Explainable Interface for Human-Autonomy Teaming: A Survey

Xiangqi Kong, Yang Xing+5 more|arXiv (Cornell University)|May 4, 2024

Business Process Modeling and Analysis6 citations

TL;DR

A comprehensive survey proposing an Explainable Interface (EI) framework for human-autonomy teaming, clarifying concepts, architecture, and evaluation from a human-centric XAI perspective.

ABSTRACT

Nowadays, large-scale foundation models are being increasingly integrated into numerous safety-critical applications, including human-autonomy teaming (HAT) within transportation, medical, and defence domains. Consequently, the inherent 'black-box' nature of these sophisticated deep neural networks heightens the significance of fostering mutual understanding and trust between humans and autonomous systems. To tackle the transparency challenges in HAT, this paper conducts a thoughtful study on the underexplored domain of Explainable Interface (EI) in HAT systems from a human-centric perspective, thereby enriching the existing body of research in Explainable Artificial Intelligence (XAI). We explore the design, development, and evaluation of EI within XAI-enhanced HAT systems. To do so, we first clarify the distinctions between these concepts: EI, explanations and model explainability, aiming to provide researchers and practitioners with a structured understanding. Second, we contribute to a novel framework for EI, addressing the unique challenges in HAT. Last, our summarized evaluation framework for ongoing EI offers a holistic perspective, encompassing model performance, human-centered factors, and group task objectives. Based on extensive surveys across XAI, HAT, psychology, and Human-Computer Interaction (HCI), this review offers multiple novel insights into incorporating XAI into HAT systems and outlines future directions.

Motivation & Objective

• Clarify the distinctions between explainable interface, explanations, and model explainability in HAT.
• Introduce a design-oriented framework for EI within HAT that covers task- and experience-oriented perspectives.
• Summarize methods for enhancing model explainability, including LLM-based explanation generation.
• Propose an evaluation framework for EI-HAT encompassing model performance, human factors, and group performance.
• Outline challenges and future directions for integrating EI into HAT across domains.

Proposed method

• Propose a stages-based framework for model explainability across pre-modeling, model design, and post-modeling phases.
• Review and synthesize literature on XAI, HAT, psychology, and HCI to ground EI-HAT concepts.
• Present an architecture for EI-HAT including interface, functional module, support systems, and human-autonomy loop.
• Discuss design principles such as user-centered automation, transparency, predictability, and mutual understanding.
• Describe how LLMs can be leveraged for explanation generation within EI-HAT.

Figure 1. Design Approach for Explainable Interface enhanced Human-Autonomy Teaming.

Experimental results

Research questions

• RQ1 What are the essential distinctions and roles of explainable interface, explanations, and model explainability in HAT?
• RQ2 How can a practical EI-HAT framework be designed to address task- and experience-oriented objectives?
• RQ3 What methods and stages best support interpretable and trustworthy explanations across pre-modeling, design, and post-modeling phases?
• RQ4 What evaluation approaches (model-based, human-centered, teaming task-focused) are appropriate for EI-HAT?
• RQ5 What challenges and future directions exist for integrating EI into HAT across healthcare, transportation, and digital twin domains?

Key findings

• EI-HAT can enhance understanding, communication, and coordination between humans and autonomous agents.
• A structured, stages-based explainability framework can guide data preparation, model design, and post-hoc explanations.
• Inherent interpretability and explainability methods should be selected per stage, with LLMs aiding explanation generation.
• An architecture with a human-autonomy loop and multimodal explanations supports mutual understanding and trust.
• Stakeholder needs and regulatory considerations (e.g., GDPR, RMF) influence EI-HAT design and evaluation.
• Future EI-HAT research should address unified terminology, design guidelines, and cross-domain applicability.

Figure 2. Clarification of Model Explainability, Explanations, Explainable Interfaces.