[Paper Review] Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey

Richard Dazeley, Peter Vamplew+1 more|arXiv (Cornell University)|Aug 20, 2021

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper proposes the Causal XRL Framework (CXF), a conceptual model that unifies Explainable Reinforcement Learning (XRL) as a backbone for Broad-XAI by integrating causal reasoning across perception, goals, actions, and outcomes. It demonstrates that XRL enables multi-level, conversational explanations beyond simple decision interpretation, positioning RL as central to building trusted, socially acceptable AI systems.

ABSTRACT

Broad Explainable Artificial Intelligence moves away from interpreting individual decisions based on a single datum and aims to provide integrated explanations from multiple machine learning algorithms into a coherent explanation of an agent's behaviour that is aligned to the communication needs of the explainee. Reinforcement Learning (RL) methods, we propose, provide a potential backbone for the cognitive model required for the development of Broad-XAI. RL represents a suite of approaches that have had increasing success in solving a range of sequential decision-making problems. However, these algorithms all operate as black-box problem solvers, where they obfuscate their decision-making policy through a complex array of values and functions. EXplainable RL (XRL) is relatively recent field of research that aims to develop techniques to extract concepts from the agent's: perception of the environment; intrinsic/extrinsic motivations/beliefs; Q-values, goals and objectives. This paper aims to introduce a conceptual framework, called the Causal XRL Framework (CXF), that unifies the current XRL research and uses RL as a backbone to the development of Broad-XAI. Additionally, we recognise that RL methods have the ability to incorporate a range of technologies to allow agents to adapt to their environment. CXF is designed for the incorporation of many standard RL extensions and integrated with external ontologies and communication facilities so that the agent can answer questions that explain outcomes and justify its decisions.

Motivation & Objective

• To establish Explainable Reinforcement Learning (XRL) as a distinct and foundational branch of Explainable AI (XAI), distinct from traditional Interpretable Machine Learning (IML).
• To address the critical gap in current XAI research, which predominantly focuses on zero-order explanations (e.g., feature importance), by advocating for multi-level, integrated explanations aligned with human cognitive and social models.
• To propose a conceptual framework—Causal XRL Framework (CXF)—that structures explanations across perception, goals, actions, and outcomes using causal relationships.
• To identify and map existing XRL techniques to the CXF components, revealing research gaps and future directions in goal-driven, emotion-aware, and hierarchical RL-based explanation.
• To position XRL not just as a tool for interpreting decisions, but as a cognitive backbone for developing truly Broad-XAI systems capable of conversational, context-aware justification of agent behavior.

Proposed method

• Proposes the Causal XRL Framework (CXF), a directed causal graph with seven components modeling how agent behavior leads to outcomes through perception, dispositions, goals, and actions.
• Adapts the Casual Explanation Network (CEN) from cognitive science and attribution theory to model human-like causal reasoning in AI agents.
• Introduces a Simplified-CXF focusing on perception and action causes, reflecting the current state of XRL research and serving as a baseline for evaluation.
• Maps existing XRL techniques to CXF components: XRL-Perception draws from IML for feature extraction and interpretability; XRL-Behavior uses introspection of Q-values, policies, and value functions to explain action sequences.
• Integrates external ontologies and communication mechanisms into the framework to enable natural language explanations tailored to explainee needs.
• Proposes future research directions such as using predictive state encoders for event-based explanations and explicability in hierarchical, multi-goal, and intrinsically motivated RL.

Experimental results

Research questions

• RQ1 How can XRL be conceptualized as a distinct and foundational component of Broad-XAI, beyond standard IML approaches?
• RQ2 What causal components are necessary to model a comprehensive explanation of an RL agent’s behavior across perception, goals, actions, and outcomes?
• RQ3 How do current XRL techniques align with the proposed Causal XRL Framework (CXF), and where are the key research gaps?
• RQ4 In what ways can hierarchical, multi-goal, and intrinsically motivated RL be leveraged to support goal-driven and emotion-aware explanations?
• RQ5 What role can predictive state representations and expectation-based reasoning play in advancing event-based and contrastive explanations in XRL?

Key findings

• The Causal XRL Framework (CXF) successfully maps the full spectrum of causal reasoning in RL agents, from perception to outcome, using a cognitive science-inspired structure with seven interconnected components.
• Most current XRL-Perception research is derived from IML techniques, particularly in function approximation and state feature interpretation, though emerging methods go beyond IML by using introspection of policy and value functions.
• XRL-Behavior research shows that temporal and sequential reasoning in RL enables causal explanations over action sequences, with model-based and value-based methods providing dynamic, interpretable explanations.
• Counterfactual and contrastive explanations are emerging in XRL by leveraging causal links between state features and outcomes, indicating a shift toward more human-aligned explanation types.
• The Simplified-CXF, focusing on perception and action causes, reflects the current state of the art and serves as a practical benchmark for evaluating XRL methods.
• Future research opportunities include integrating hierarchical RL, multi-objective optimization, intrinsic motivation, and emotion-aware models into the CXF to support richer, context-sensitive explanations.