[论文解读] GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanations

Tobias Huber, Maximilian Demmler+3 more|arXiv (Cornell University)|Feb 24, 2023

Explainable Artificial Intelligence (XAI)被引用 4

一句话总结

该论文提出 GANterfactual-RL，一种用于深度强化学习中生成视觉反事实解释的模型无关方法，通过将反事实生成建模为基于 StarGAN 的对抗性学习的域迁移问题。该方法在计算指标上优于先前工作，并在用户研究中显著提升了用户对强化学习智能体策略的理解，尽管由于伪影和展示问题，主观满意度仍较低。

ABSTRACT

Counterfactual explanations are a common tool to explain artificial intelligence models. For Reinforcement Learning (RL) agents, they answer "Why not?" or "What if?" questions by illustrating what minimal change to a state is needed such that an agent chooses a different action. Generating counterfactual explanations for RL agents with visual input is especially challenging because of their large state spaces and because their decisions are part of an overarching policy, which includes long-term decision-making. However, research focusing on counterfactual explanations, specifically for RL agents with visual input, is scarce and does not go beyond identifying defective agents. It is unclear whether counterfactual explanations are still helpful for more complex tasks like analyzing the learned strategies of different agents or choosing a fitting agent for a specific task. We propose a novel but simple method to generate counterfactual explanations for RL agents by formulating the problem as a domain transfer problem which allows the use of adversarial learning techniques like StarGAN. Our method is fully model-agnostic and we demonstrate that it outperforms the only previous method in several computational metrics. Furthermore, we show in a user study that our method performs best when analyzing which strategies different agents pursue.

研究动机与目标

• 为具有视觉输入的深度强化学习智能体解决缺乏有效、可视化反事实解释的问题。
• 通过回答“如果……会怎样？”或“为什么不……？”等问题，使用户能够理解不同强化学习智能体的潜在策略。
• 开发一种模型无关的方法，生成最小且逼真的输入扰动以改变智能体的决策。
• 评估反事实解释是否能超越现有解释技术，提升用户对智能体行为的理解。
• 识别当前反事实解释的局限性，特别是用户满意度和信任校准方面的问题。

提出的方法

• 该方法将反事实生成建模为域迁移问题，将原始状态转换为能引发不同动作的反事实状态。
• 利用基于 StarGAN 的生成对抗网络执行域迁移，支持端到端训练，结合对抗损失和重建损失。
• 该方法完全模型无关，可应用于任何预训练的强化学习智能体，无需访问策略网络或奖励函数。
• 生成器被训练以最小化动作变化，同时保持感知真实性和最小化输入扰动。
• 该方法使用判别器确保生成的反事实在视觉上逼真，难以与真实状态区分。
• 该方法可生成复杂、高维视觉输入（如 Atari 游戏画面）的反事实。

Figure 1. Example for a counterfactual explanation: In the original situation (a), the agent does not take the fastest path to the pill in the top right corner. It is unclear if the agent is afraid of the ghost or does not recognize the shortest path. The counterfactual state (b) shows that the agen

实验结果

研究问题

• RQ1 反事实解释能否有效揭示具有视觉输入的深度强化学习智能体的决策策略？
• RQ2 与强化学习中唯一先前的视觉反事实方法相比，GANterfactual-RL 在计算效率和保真度方面表现如何？
• RQ3 与显著性图或无解释相比，反事实解释在多大程度上提升了用户对智能体行为的理解？
• RQ4 尽管客观理解能力提升，为何用户对反事实解释的满意度反而低于无解释？
• RQ5 反事实解释能否帮助用户为特定任务选择最合适的智能体？

主要发现

• GANterfactual-RL 在所有计算指标上均优于先前最先进方法，包括成功率、扰动幅度和推理时间。
• 在用户研究中，使用 GANterfactual-RL 的参与者在智能体理解任务中得分达到 50%，显著高于使用显著性图的 37%。
• 尽管客观理解能力提升，用户对反事实解释的满意度仍低于无解释，表明存在主观可用性差距。
• 参与者指出反事实中存在视觉伪影，影响了感知质量和信任度。
• 反事实解释并未显著提升用户选择最适合任务的智能体的能力，因为这需要超越策略理解的领域专业知识。
• 研究证实，反事实更有利于用户完善对智能体行为的心理模型，而非校准对智能体的信任。

Figure 2. Schematic of our counterfactual generation approach. We formulate the problem as domain transfer where each domain represents an action. States are assigned to domains based on the action that the agent chooses for them.