[Paper Review] Explainable Autonomous Robots: A Survey and Perspective

Tatsuya Sakai, Takayuki Nagai|arXiv (Cornell University)|May 6, 2021

Explainable Artificial Intelligence (XAI)63 references4 citations

TL;DR

This survey proposes a framework for explainable autonomous robots (XAR) that enables robots to dynamically explain their actions, intentions, and decisions in human-understandable terms by modeling user mental states and adjusting explanation granularity. The key contribution is a unified definition of explainability in autonomous systems and a roadmap for integrating explanation generation into robot decision-making through adaptive, context-aware communication.

ABSTRACT

Advanced communication protocols are critical to enable the coexistence of autonomous robots with humans. Thus, the development of explanatory capabilities is an urgent first step toward autonomous robots. This survey provides an overview of the various types of "explainability" discussed in machine learning research. Then, we discuss the definition of "explainability" in the context of autonomous robots (i.e., explainable autonomous robots) by exploring the question "what is an explanation?" We further conduct a research survey based on this definition and present some relevant topics for future research.

Motivation & Objective

• To define explainability in the context of autonomous robots beyond machine learning interpretability, focusing on mutual understanding between humans and robots.
• To address the critical gap in human-robot trust due to opaque decision-making, especially in real-time interactions.
• To propose a framework where robots estimate user mental models and tailor explanations based on user knowledge and trust level.
• To identify core research challenges in explanation timing, content, and ethical integrity in robot communication.
• To integrate explanation generation into the robot's action space as a first-class component of autonomous behavior.

Proposed method

• Proposes a definition of explainability in autonomous robots as the ability to convey reasoning behind actions in a way that aligns with human cognitive models.
• Introduces a three-state world model: robot's internal state, user's internal state, and external environment, to enable explanation generation.
• Treats explanation presentation as an action in the robot's action space, enabling learning via standard reinforcement learning algorithms.
• Uses model comparison between user’s predicted behavior and robot’s own decision model to extract relevant explanation content.
• Employs adaptive explanation granularity based on user-specific trust and knowledge levels to avoid information overload.
• Considers ethical implications of non-truthful explanations, proposing that lying undermines long-term trust despite short-term utility.

Experimental results

Research questions

• RQ1 What constitutes a valid explanation in the context of autonomous robots, and how does it differ from interpretability in machine learning?
• RQ2 How can a robot dynamically estimate the user’s mental model to generate contextually appropriate explanations?
• RQ3 What mechanisms allow robots to decide when, what, and how to explain during human-robot interaction?
• RQ4 How can explanation generation be formally integrated into the robot’s decision-making and action selection process?
• RQ5 What are the ethical boundaries of generating explanations that deviate from actual reasoning processes?

Key findings

• Explainability in autonomous robots is fundamentally a communication problem requiring mutual understanding of internal states, not just output explanation.
• The granularity and content of explanations must be tailored to the user’s knowledge and trust level to avoid overwhelming or misleading the user.
• Explanation generation can be modeled as an action in the robot’s action space, enabling end-to-end learning with standard RL frameworks.
• Failure to explain leads to distrust, and even accurate explanations can be ineffective if poorly timed or irrelevant to the user’s concerns.
• Ethical concerns arise when robots generate explanations that are factually inaccurate, as such practices risk eroding long-term trust in the system.
• The framework enables robots to convey not only 'what' they are doing but also 'why' they are doing it, fostering a sense of shared intentionality.