[Paper Review] State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding

Devleena Das, Sonia Chernova+1 more|arXiv (Cornell University)|Sep 21, 2023

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper proposes State2Explanation (S2E), a unified framework that learns a joint embedding between agent states/actions and concept-based explanations to simultaneously improve reinforcement learning agent training via reward shaping and enhance end-user understanding at deployment. In Connect 4 and Lunar Lander, S2E significantly boosts learning efficiency and user task performance, demonstrating dual benefits through concept-based explanations grounded in goal-oriented abstractions.

ABSTRACT

As more non-AI experts use complex AI systems for daily tasks, there has been an increasing effort to develop methods that produce explanations of AI decision making that are understandable by non-AI experts. Towards this effort, leveraging higher-level concepts and producing concept-based explanations have become a popular method. Most concept-based explanations have been developed for classification techniques, and we posit that the few existing methods for sequential decision making are limited in scope. In this work, we first contribute a desiderata for defining concepts in sequential decision making settings. Additionally, inspired by the Protege Effect which states explaining knowledge often reinforces one's self-learning, we explore how concept-based explanations of an RL agent's decision making can in turn improve the agent's learning rate, as well as improve end-user understanding of the agent's decision making. To this end, we contribute a unified framework, State2Explanation (S2E), that involves learning a joint embedding model between state-action pairs and concept-based explanations, and leveraging such learned model to both (1) inform reward shaping during an agent's training, and (2) provide explanations to end-users at deployment for improved task performance. Our experimental validations, in Connect 4 and Lunar Lander, demonstrate the success of S2E in providing a dual-benefit, successfully informing reward shaping and improving agent learning rate, as well as significantly improving end user task performance at deployment time.

Motivation & Objective

• To address the gap in concept-based explanations for sequential decision-making systems, especially beyond preconditions and action costs.
• To investigate whether concept-based explanations can benefit both the AI agent (via reward shaping) and the end-user (via improved understanding).
• To develop a unified framework that learns joint embeddings between states/actions and explanations for dual-use applications.
• To define a desiderata for what constitutes a 'concept' in sequential decision-making, emphasizing high-level, goal-relevant abstractions.
• To validate S2E’s effectiveness in improving both agent learning rates and end-user task performance in complex RL environments.

Proposed method

• S2E learns a joint embedding model between state-action pairs and concept-based explanations using a contrastive learning objective.
• The learned embedding enables automatic reward shaping during agent training by identifying when and how to apply concept-based rewards.
• At deployment, the model retrieves concept-based explanations for user interpretation using the same embedding space.
• The framework incorporates two abstraction techniques: InF (Influence-based Filtering) and TeG (Task-Goal abstraction) to generate high-level, user-friendly explanations.
• Concepts are defined based on goal-relevant influence, not just preconditions or logical formulas, following a proposed desiderata for sequential decision-making.
• The method is evaluated using both model-based metrics (learning rate, sample efficiency) and user studies (task performance, ATS) in Connect 4 and Lunar Lander.

Experimental results

Research questions

• RQ1 Can concept-based explanations improve the learning rate of an RL agent through reward shaping?
• RQ2 Can concept-based explanations enhance end-user understanding and task performance in sequential decision-making tasks?
• RQ3 What defines a 'concept' in the context of sequential decision-making beyond preconditions and action costs?
• RQ4 How do abstraction techniques like InF and TeG affect the effectiveness of concept-based explanations for end-users?
• RQ5 Can a unified embedding model effectively support both agent training and user explanation retrieval?

Key findings

• S2E achieved learning rates comparable to expert-defined reward functions in both Connect 4 and Lunar Lander, demonstrating effective automatic reward shaping.
• In user evaluations, exposure to S2E-generated explanations significantly improved participant task performance (ATS) compared to no explanation, natural language explanations, and visual-only explanations.
• The combination of InF and TeG abstraction methods in S2E led to the most significant improvement in user task performance, particularly in high-frequency RL domains like Lunar Lander.
• User studies showed a statistically significant improvement in ATS when participants received S2E explanations with both InF and TeG abstractions (p < 0.01), outperforming all baselines.
• The joint embedding model successfully retrieved concept-based explanations that were both semantically meaningful and useful for both agent training and user understanding.
• Incorrect explanation retrievals had minimal impact on performance in tested domains, but the risk of misleading explanations in high-stakes applications remains a concern for future work.