[Paper Review] Causal Confusion in Imitation Learning

Pim de Haan, Dinesh Jayaraman+1 more|arXiv (Cornell University)|May 28, 2019

Robot Manipulation and Learning56 references127 citations

TL;DR

The paper identifies causal misidentification as a core failure mode in imitation learning under distributional shift and proposes a causally grounded intervention framework to recover the true causal model via targeted environment interactions or expert queries.

ABSTRACT

Behavioral cloning reduces policy learning to supervised learning by training a discriminative model to predict expert actions given observations. Such discriminative models are non-causal: the training procedure is unaware of the causal structure of the interaction between the expert and the environment. We point out that ignoring causality is particularly damaging because of the distributional shift in imitation learning. In particular, it leads to a counter-intuitive "causal misidentification" phenomenon: access to more information can yield worse performance. We investigate how this problem arises, and propose a solution to combat it through targeted interventions---either environment interaction or expert queries---to determine the correct causal model. We show that causal misidentification occurs in several benchmark control domains as well as realistic driving settings, and validate our solution against DAgger and other baselines and ablations.

Motivation & Objective

• Identify how non-causal behavior cloning can fail under distributional shift due to causal misidentification.
• Formalize the role of causal graphs and interventions in imitation learning.
• Propose a graph-parameterized policy framework and targeted interventions to recover the correct causal model.
• Evaluate the approach against standard baselines and ablations on benchmark control tasks and realistic driving scenarios.

Proposed method

• Model policies as mixtures parameterized by a causal graph G, where each graph defines whether a state feature is a cause of the expert action.
• Train a graph-parameterized policy by sampling graphs and minimizing action prediction loss across demonstrations.
• Use interventional queries (expert or environment-based) to score and identify the true causal graph by evaluating disagreement or returns.
• Provide two intervention modes: expert query interventions (active querying on informative states) and policy execution interventions (compare episode returns across graphs).
• Disentangle observations with a beta-VAE to ensure state representations separate causes from nuisances when using image data.

Experimental results

Research questions

• RQ1 What is causal misidentification in imitation learning and how does it arise under distributional shift?
• RQ2 Can targeted interventions identify the true causal model and improve robustness of imitation policies?
• RQ3 How effective are expert queries versus environment-based interventions in recovering the correct graph?
• RQ4 Does disentangling observations influence the ability to discover true causes?
• RQ5 How do the proposed methods compare to DAgger, dropout, and GAIL on standard benchmarks and realistic driving tasks?

Key findings

• Causal misidentification leads to worse imitation performance when more information is available, due to nuisance correlations misguiding the policy.
• Graph-parameterized policies plus targeted interventions can identify the correct causal model and close the performance gap.
• Policy-execution interventions typically converge to the true causal model with fewer episodes than expert-query approaches in driving-like tasks.
• Expert-query interventions require fewer queries than DAgger to achieve comparable improvements in several tasks.
• Disentangled state representations (via beta-VAE) are important for effectively discovering true causes, improving intervention outcomes.