[Paper Review] Causal Reasoning from Meta-reinforcement Learning

Ishita Dasgupta, Jane X. Wang+8 more|arXiv (Cornell University)|Jan 23, 2019

Bayesian Modeling and Causal InferenceComputer Science6 references74 citations

TL;DR

The paper shows that a model-free, meta-learned RNN agent can perform causal reasoning—do-calculus, intervention, and counterfactuals—across observational, interventional, and counterfactual data settings by learning to design informative experiments.

ABSTRACT

Discovering and exploiting the causal structure in the environment is a crucial challenge for intelligent agents. Here we explore whether causal reasoning can emerge via meta-reinforcement learning. We train a recurrent network with model-free reinforcement learning to solve a range of problems that each contain causal structure. We find that the trained agent can perform causal reasoning in novel situations in order to obtain rewards. The agent can select informative interventions, draw causal inferences from observational data, and make counterfactual predictions. Although established formal causal reasoning algorithms also exist, in this paper we show that such reasoning can arise from model-free reinforcement learning, and suggest that causal reasoning in complex settings may benefit from the more end-to-end learning-based approaches presented here. This work also offers new strategies for structured exploration in reinforcement learning, by providing agents with the ability to perform -- and interpret -- experiments.

Motivation & Objective

• Motivate whether causal reasoning can emerge from meta-learning without explicit causal priors.
• Demonstrate causal inference capabilities (causal effects, interventions, counterfactuals) from observational data.
• Show that active data collection improves causal understanding and task rewards.
• Evaluate whether meta-learned agents can transfer to novel causal graphs not seen during training.

Proposed method

• Train an LSTM-based agent with model-free reinforcement learning to act on random causal Bayesian networks (CBNs).
• Structure episodes into information and quiz phases where interventions or observations are used to infer causal structure.
• Use do-calculus-inspired reasoning to derive causal effects in observational data and test against counterfactual reasoning tasks.
• Compare active (informed) vs random data collection to assess the value of structured exploration.
• Evaluate across three settings: observational, interventional, and counterfactual with held-out test graphs.

Experimental results

Research questions

• RQ1 Can a meta-trained, model-free RL agent perform causal reasoning from observational data alone?
• RQ2 Does access to interventional data enable solving causal questions in the presence of unobserved confounders?
• RQ3 Can agents perform counterfactual reasoning, and does abduction improve performance in degenerate cases?
• RQ4 Do agents learn to actively select informative observations or interventions to improve quiz-phase rewards?
• RQ5 How well do learned strategies transfer to unseen causal graphs?

Key findings

• Agents trained with meta-learning can perform cause-effect reasoning from observational data, outperforming the best purely associative baseline when the intervened node has parents.
• Interventional data allows resolving unobserved confounders, with active interventional agents outperforming observational-only agents in confounded cases.
• Counterfactual agents, leveraging latent randomness (abduction), exceed interventional agents in degenerate maximum scenarios and when tailored interventions are used.
• Active data collection policies yield higher quiz-phase rewards than random observation policies across all three experimental settings.
• Agents demonstrate learning-to-do-calculus, effective intervention planning, and counterfactual prediction without explicit causal priors.