[Paper Review] GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanations

Tobias Huber, Maximilian Demmler+3 more|arXiv (Cornell University)|Feb 24, 2023

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper proposes GANterfactual-RL, a model-agnostic method for generating visual counterfactual explanations in deep reinforcement learning by framing counterfactual generation as a domain translation problem using StarGAN-based adversarial learning. It outperforms prior work in computational metrics and significantly improves users' understanding of RL agents' strategies in a user study, though subjective satisfaction remains low due to artifacts and presentation issues.

ABSTRACT

Counterfactual explanations are a common tool to explain artificial intelligence models. For Reinforcement Learning (RL) agents, they answer "Why not?" or "What if?" questions by illustrating what minimal change to a state is needed such that an agent chooses a different action. Generating counterfactual explanations for RL agents with visual input is especially challenging because of their large state spaces and because their decisions are part of an overarching policy, which includes long-term decision-making. However, research focusing on counterfactual explanations, specifically for RL agents with visual input, is scarce and does not go beyond identifying defective agents. It is unclear whether counterfactual explanations are still helpful for more complex tasks like analyzing the learned strategies of different agents or choosing a fitting agent for a specific task. We propose a novel but simple method to generate counterfactual explanations for RL agents by formulating the problem as a domain transfer problem which allows the use of adversarial learning techniques like StarGAN. Our method is fully model-agnostic and we demonstrate that it outperforms the only previous method in several computational metrics. Furthermore, we show in a user study that our method performs best when analyzing which strategies different agents pursue.

Motivation & Objective

• To address the lack of effective, visual counterfactual explanations for deep reinforcement learning agents with visual input.
• To enable users to understand the underlying strategies of different RL agents by answering 'What if?' or 'Why not?' questions.
• To develop a model-agnostic method that generates minimal, realistic input perturbations to change agent decisions.
• To evaluate whether counterfactual explanations improve user understanding of agent behavior beyond existing explanation techniques.
• To identify limitations in current counterfactual explanations, particularly in user satisfaction and trust calibration.

Proposed method

• The method formulates counterfactual generation as a domain transfer problem, where the original state is transformed into a counterfactual state that induces a different action.
• It leverages StarGAN-based generative adversarial networks to perform the domain translation, enabling end-to-end training with adversarial loss and reconstruction loss.
• The approach is fully model-agnostic, allowing application to any pre-trained RL agent without requiring access to the policy network or reward function.
• The generator is trained to minimize action changes while preserving perceptual realism and minimizing input perturbations.
• The method uses a discriminator to ensure generated counterfactuals are realistic and indistinguishable from real states.
• It enables the generation of counterfactuals for complex, high-dimensional visual inputs such as Atari game frames.

Figure 1. Example for a counterfactual explanation: In the original situation (a), the agent does not take the fastest path to the pill in the top right corner. It is unclear if the agent is afraid of the ghost or does not recognize the shortest path. The counterfactual state (b) shows that the agen

Experimental results

Research questions

• RQ1 Can counterfactual explanations effectively reveal the decision-making strategies of deep RL agents with visual input?
• RQ2 How does GANterfactual-RL compare to the only prior method for visual counterfactuals in RL in terms of computational efficiency and fidelity?
• RQ3 To what extent do counterfactual explanations improve users' understanding of agent behavior compared to saliency maps or no explanation?
• RQ4 Why do users report lower satisfaction with counterfactual explanations despite improved objective understanding?
• RQ5 Can counterfactual explanations help users select the most suitable agent for a given task?

Key findings

• GANterfactual-RL outperformed the prior state-of-the-art method in all computational metrics, including success rate, perturbation magnitude, and inference time.
• In a user study, participants using GANterfactual-RL achieved a 50% score on the agent understanding task, significantly outperforming the 37% score with saliency maps.
• Despite improved objective understanding, users reported lower satisfaction with counterfactual explanations than with no explanation, indicating a subjective usability gap.
• Participants noted that counterfactuals contained visual artifacts, which negatively impacted perceived quality and trust.
• Counterfactual explanations did not significantly improve users’ ability to select the best agent for a task, as this required domain-specific expertise beyond strategy understanding.
• The study confirmed that counterfactuals are more effective for refining mental models of agent behavior than for calibrating trust in agents.

Figure 2. Schematic of our counterfactual generation approach. We formulate the problem as domain transfer where each domain represents an action. States are assigned to domains based on the action that the agent chooses for them.