[Paper Review] Causal Imitation Learning with Unobserved Confounders

Junzhe Zhang, Daniel Kumor+1 more|arXiv (Cornell University)|Aug 12, 2022

Reinforcement Learning in Robotics31 references23 citations

TL;DR

The paper develops a causal framework for imitation learning when the demonstrator and learner observe different covariates due to unobserved confounders, providing graphical criteria and practical algorithms to learn imitating policies from demonstrations.

ABSTRACT

One of the common ways children learn is by mimicking adults. Imitation learning focuses on learning policies with suitable performance from demonstrations generated by an expert, with an unspecified performance measure, and unobserved reward signal. Popular methods for imitation learning start by either directly mimicking the behavior policy of an expert (behavior cloning) or by learning a reward function that prioritizes observed expert trajectories (inverse reinforcement learning). However, these methods rely on the assumption that covariates used by the expert to determine her/his actions are fully observed. In this paper, we relax this assumption and study imitation learning when sensory inputs of the learner and the expert differ. First, we provide a non-parametric, graphical criterion that is complete (both necessary and sufficient) for determining the feasibility of imitation from the combinations of demonstration data and qualitative assumptions about the underlying environment, represented in the form of a causal model. We then show that when such a criterion does not hold, imitation could still be feasible by exploiting quantitative knowledge of the expert trajectories. Finally, we develop an efficient procedure for learning the imitating policy from experts' trajectories.

Motivation & Objective

• Motivate imitation learning when expert inputs are affected by unobserved covariates and rewards are latent.
• Provide a complete graphical criterion to assess imitability from a causal diagram and observational data.
• Develop a sufficient algorithm to identify an imitating policy when imitability does not hold identifiably.
• Offer practical procedures for learning imitating policies via explicit causal parametrization and validation on synthetic data.

Proposed method

• Introduce partially observable structural causal models (POSCMs) to model observed and latent endogenous variables.
• Define identifiability and imitatability notions for policies with latent rewards.
• Prove Global and backdoor-based criteria (Imitation by Direct Parents and Imitation by π-Backdoor) for when imitation is feasible.
• Introduce the Imitate algorithm that searches for imitation instruments (surrogates and identifiable subspaces) and learns policies by solving P(s|do(π)) = P(s).
• Provide a practical optimization framework where P(s|do(π)) is identifiable within an identified subspace and solved via standard density estimation or linear equation systems.
• Outline procedures to select surrogates, identify instruments, and implement optimization to obtain an imitating policy.

Experimental results

Research questions

• RQ1 Under what graphical conditions is imitating the expert’s reward feasible when rewards are latent and confounding is present?
• RQ2 How can one leverage backdoor-like criteria and observational data to construct imitating policies when the expert’s policy lies outside the learner’s policy space?
• RQ3 How can surrogate variables and identifiable subspaces be used to learn policies that replicate expert performance in POSCMs?
• RQ4 What practical algorithms can efficiently find imitating policies using real-valued observational distributions?

Key findings

• A complete graphical criterion (Imitation by Direct Parents) identifies when imitation is feasible based on the causal graph and policy space.
• A second criterion (Imitation by π-Backdoor) characterizes imitatability via backdoor-admissible sets that enable policy-based imitation using observed data.
• An extended framework (practical imitability) shows imitation can be achieved by leveraging actual observational distributions P(o) and surrogate variables even when pure identifiability fails.
• Introduction of the Imitate algorithm to search for imitation instruments and identifiable subspaces and to compute a policy that satisfies P(s|do(π)) = P(s).
• Proof-of-concept: the approach provides a practical method for learning imitating policies via parametric/casual modeling and validation on high-dimensional synthetic datasets.