[论文解读] Distal Explanations for Explainable Reinforcement Learning Agents

Prashan Madumal, Tim Miller+2 more|arXiv (Cornell University)|Jan 28, 2020

Explainable Artificial Intelligence (XAI)参考文献 22被引用 13

一句话总结

该论文提出了一种远端解释模型，通过将因果模型与决策树结合并引入循环神经网络，为可解释强化学习生成反事实和机会链——即类似'A促成B，B导致C'的因果链——的解释。在六个强化学习基准和包含90名参与者的用户研究中，与基线解释模型相比，该模型在对抗性、搜救和协作场景中均提升了任务预测准确率和用户理解能力。

ABSTRACT

Causal explanations present an intuitive way to understand the course of events through causal chains, and are widely accepted in cognitive science as the prominent model humans use for explanation. Importantly, causal models can generate opportunity chains, which take the form of `A enables B and B causes C'. We ground the notion of opportunity chains in human-agent experimental data, where we present participants with explanations from different models and ask them to provide their own explanations for agent behaviour. Results indicate that humans do in-fact use the concept of opportunity chains frequently for describing artificial agent behaviour. Recently, action influence models have been proposed to provide causal explanations for model-free reinforcement learning (RL). While these models can generate counterfactuals---things that did not happen but could have under different conditions---they lack the ability to generate explanations of opportunity chains. We introduce a distal explanation model that can analyse counterfactuals and opportunity chains using decision trees and causal models. We employ a recurrent neural network to learn opportunity chains and make use of decision trees to improve the accuracy of task prediction and the generated counterfactuals. We computationally evaluate the model in 6 RL benchmarks using different RL algorithms, and show that our model performs better in task prediction. We report on a study with 90 participants who receive explanations of RL agents behaviour in solving three scenarios: 1) Adversarial; 2) Search and rescue; and 3) Human-Agent collaborative scenarios. We investigate the participants' understanding of the agent through task prediction and their subjective satisfaction of the explanations and show that our distal explanation model results in improved outcomes over the three scenarios compared with two baseline explanation models.

研究动机与目标

• 解决现有无模型强化学习动作影响模型中缺乏机会链解释的问题。
• 通过建模符合人类认知解释模式的因果链，提升人类对强化学习智能体行为的理解。
• 构建一个统一框架，结合反事实推理与机会链生成，利用决策树和RNN实现。
• 在多样化强化学习场景中，通过计算基准和人机协同研究评估模型的有效性。

提出的方法

• 该模型使用循环神经网络从智能体行为序列中学习并生成机会链。
• 采用决策树优化任务预测并提升生成反事实的准确性。
• 集成因果模型以分析和结构化生成的机会链中的因果关系。
• 将反事实生成与机会链建模相结合，生成更直观、符合人类认知的解释。
• 通过多种强化学习算法在六个强化学习基准上进行计算评估，以衡量任务预测性能。
• 通过包含90名参与者的用户研究，评估三种场景（对抗性、搜救、人机协作）中解释质量的表现。

实验结果

研究问题

• RQ1 能否以'A促成B且B导致C'的形式结构化机会链，有效建模人类对强化学习智能体行为的解释？
• RQ2 将机会链与反事实推理结合，能否提升强化学习智能体的任务预测准确率？
• RQ3 与基线解释模型相比，所提出的远端解释模型是否能增强人类用户对智能体行为的理解？
• RQ4 在多样化强化学习场景中，用户对本模型生成的解释是否表现出更高的满意度与信息量？

主要发现

• 与基线模型相比，该远端解释模型在六个强化学习基准上显著提升了任务预测准确率。
• 在所有三种场景中，接触本模型生成解释的参与者均表现出更优的任务预测表现。
• 在人机协作研究中，用户对本模型生成解释的主观满意度高于两种基线模型。
• 通过参与者提供的解释验证，该模型成功生成了与人类认知解释模式一致的机会链。
• 决策树的引入显著提升了模型中反事实生成与任务预测的准确性。
• 在计算评估与人类评估中，该模型均优于基线模型，展现出在多样化强化学习环境中的稳健性。