[论文解读] Batch Inverse Reinforcement Learning Using Counterfactuals for Understanding Decision Making.

Ioana Bica, Daniel Jarrett+2 more|arXiv (Cornell University)|Jul 2, 2020

Health Systems, Economic Evaluations, Quality of Life参考文献 37被引用 2

一句话总结

本文提出了一种批量逆强化学习方法，通过整合反事实推理，从示范轨迹中解释专家决策。通过在每个决策点回答“如果会怎样”的问题，该方法学习可解释的奖励函数，并在无需主动交互的情况下实现离线策略评估，展示了在医疗决策环境中的优异性能。

ABSTRACT

A key challenge in modeling real-world decision-making is the fact that active experimentation is often impossible (e.g. in healthcare). The goal of batch inverse reinforcement learning is to recover and understand policies on the basis of demonstrated behaviour--i.e. trajectories of observations and actions made by an expert maximizing some unknown reward function. We propose incorporating counterfactual reasoning into modeling decision behaviours in this setting. At each decision point, counterfactuals answer the question: Given the current history of observations, what would happen if we took a particular action? First, this offers a principled approach to learning inherently interpretable reward functions, which enables understanding the cost-benefit tradeoffs associated with an expert's actions. Second, by estimating the effects of different actions, counterfactuals readily tackle the off-policy nature of policy evaluation in the batch setting. Not only does this alleviate the cold-start problem typical of conventional solutions, but also accommodates settings where the expert policies are depending on histories of observations rather than just current states. Through experiments in both real and simulated medical environments, we illustrate the effectiveness of our batch, counterfactual inverse reinforcement learning approach in recovering accurate and interpretable descriptions of expert behaviour.

研究动机与目标

• 为解决在无法进行主动实验（如医疗领域）时理解专家策略的挑战。
• 在无在线交互的情况下，从静态专家轨迹数据集中建模决策过程。
• 通过整合反事实推理，提高恢复的奖励函数的可解释性。
• 通过估计动作后果，克服批量逆强化学习中的离线策略评估问题。
• 支持依赖于观察历史而非仅当前状态的策略。

提出的方法

• 将反事实推理整合到批量逆强化学习中，以评估每个决策点的假设性动作。
• 利用反事实推理，估计在给定当前观察历史条件下，替代动作的后果。
• 通过建模动作干预的影响，学习反映成本-收益权衡的奖励函数。
• 通过模拟观察轨迹中的动作变更，采用结构化方法实现离线策略评估。
• 将专家策略建模为依赖于完整观察历史，而非仅当前状态。
• 结合轨迹数据与反事实模拟，推断出可解释且准确的奖励函数。

实验结果

研究问题

• RQ1 反事实推理在批量逆强化学习中如何提升奖励函数的可解释性？
• RQ2 反事实方法能否有效解决静态专家示范数据中的离线策略评估挑战？
• RQ3 当动作依赖于历史观测时，该方法在多大程度上能恢复专家决策策略？
• RQ4 反事实建模在多大程度上增强了对专家行为中成本-收益权衡的理解？
• RQ5 该方法是否能推广到医疗等现实世界复杂领域？

主要发现

• 该方法成功恢复了反映专家决策中合理成本-收益权衡的可解释奖励函数。
• 反事实推理使无需在线交互或探索即可实现准确的离线策略评估成为可能。
• 该方法能有效建模依赖于观察历史的策略，而不仅依赖于当前状态。
• 在模拟和真实医疗环境中的实验表明，专家行为建模的准确性得到提升。
• 反事实集成减少了传统批量逆强化学习方法中常见的冷启动问题。
• 该模型在专家行为复杂且依赖历史的环境中表现出稳健的性能。