[论文解读] Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models

Michael Oberst, David Sontag|arXiv (Cornell University)|May 14, 2019

Health Systems, Economic Evaluations, Quality of Life被引用 58

一句话总结

本文提出一种基于 Gumbel-Max 结构化因果模型的反事实离策略评估框架，以在有限的 POMDP 中生成反事实轨迹，便于检查学习的 RL 策略可能在哪些方面与观测结果偏离。它通过一个合成的败血症管理环境来调试高风险策略来演示该方法。

ABSTRACT

We introduce an off-policy evaluation procedure for highlighting episodes where applying a reinforcement learned (RL) policy is likely to have produced a substantially different outcome than the observed policy. In particular, we introduce a class of structural causal models (SCMs) for generating counterfactual trajectories in finite partially observable Markov Decision Processes (POMDPs). We see this as a useful procedure for off-policy "debugging" in high-risk settings (e.g., healthcare); by decomposing the expected difference in reward between the RL and observed policy into specific episodes, we can identify episodes where the counterfactual difference in reward is most dramatic. This in turn can be used to facilitate review of specific episodes by domain experts. We demonstrate the utility of this procedure with a synthetic environment of sepsis management.

研究动机与目标

• Motivate counterfactual analysis to identify episodes where an RL policy could produce dramatically different outcomes than the observed policy.
• Develop a structural causal modeling framework enabling counterfactual trajectory generation in finite POMDPs.
• Introduce counterfactual stability and a Gumbel-Max SCM to address non-identifiability in discrete transitions.
• Provide a Monte Carlo method to sample counterfactual trajectories under the Gumbel-Max SCM.
• Demonstrate introspection capability by applying the method to a synthetic sepsis management environment for debugging.

提出的方法

• Formulate counterfactual decomposition of expected reward to highlight differences across episodes.
• Define counterfactual stability for categorical variables and prove its relation to monotonicity in the binary case.
• Introduce Gumbel-Max SCM where discrete outcomes are generated via Gumbel-max sampling and prove it satisfies counterfactual stability.
• Show how to draw counterfactual trajectories post-hoc using posterior sampling of Gumbel variables given observed outcomes.
• Provide two procedures for posterior inference: rejection sampling and a shifted-Gumbel-based sampling method for counterfactuals under intervention.

实验结果

研究问题

• RQ1 Can counterfactual trajectories be efficiently generated under a categorical SCM to diagnose RL policies in POMDPs?
• RQ2 Does counterfactual stability ensure identifiability or align with monotonicity in binary cases?
• RQ3 How can Gumbel-Max SCMs be used to draw counterfactual trajectories given observed data and a target policy?
• RQ4 What is the value of counterfactual off-policy evaluation for debugging high-risk RL applications like sepsis treatment?

主要发现

• Counterfactual decomposition allows attributing differences in rewards to specific episodes via counterfactual trajectories.
• Counterfactual stability is introduced for categorical variables and implies monotonicity in the binary case.
• Gumbel-Max SCMs satisfy counterfactual stability and enable post-hoc sampling of counterfactual trajectories.
• A Monte Carlo posterior over counterfactuals can be drawn via rejection sampling or using shifted Gumbel distributions.
• In a sepsis-inspired synthetic environment, the method reveals dangerous assumptions in the learned policy that off-policy estimates may miss.