[Paper Review] InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations

Yunzhu Li, Jiaming Song+1 more|arXiv (Cornell University)|Mar 26, 2017

Reinforcement Learning in RoboticsComputer Science39 references141 citations

TL;DR

InfoGAIL extends GAIL with a latent variable to uncover and disentangle latent factors in expert demonstrations, enabling interpretable imitation from visual inputs and learning multiple modes of behavior without supervision.

ABSTRACT

The goal of imitation learning is to mimic expert behavior without access to an explicit reward signal. Expert demonstrations provided by humans, however, often show significant variability due to latent factors that are typically not explicitly modeled. In this paper, we propose a new algorithm that can infer the latent structure of expert demonstrations in an unsupervised way. Our method, built on top of Generative Adversarial Imitation Learning, can not only imitate complex behaviors, but also learn interpretable and meaningful representations of complex behavioral data, including visual demonstrations. In the driving domain, we show that a model learned from human demonstrations is able to both accurately reproduce a variety of behaviors and accurately anticipate human actions using raw visual inputs. Compared with various baselines, our method can better capture the latent structure underlying expert demonstrations, often recovering semantically meaningful factors of variation in the data.

Motivation & Objective

• Motivate imitation learning when expert demonstrations exhibit latent variation across multiple strategies.
• Extend GAIL to discover and disentangle latent factors of variation in demonstrations.
• Enable learning from raw visual inputs and recover semantically meaningful latent structure.

Proposed method

• Augment GAIL with a latent variable c that selects among a mixture of expert policies π_E = {π_E^0, π_E^1, ...}.
• Introduce a mutual information regularization term I(c; τ) approximated by a variational bound L_I(π,Q) to force c to capture salient factors in trajectories.
• Optimize the InfoGAIL objective: min_π,Q max_D E_π[log D(s,a)] + E_{π_E}[log(1−D(s,a))] − λ1 L_I(π,Q) − λ2 H(π).
• Use a simplified posterior Q(c|s,a) to avoid expensive trajectory-level computation.
• Enhance optimization with Wasserstein GAN objectives, reward augmentation, variance-reduction techniques, and TRPO for policy updates.

Experimental results

Research questions

• RQ1 Can latent variables in the policy capture and disentangle latent variations in expert demonstrations?
• RQ2 Can InfoGAIL identify semantically meaningful factors of variation (e.g., driving styles) from visual inputs without supervision?
• RQ3 Does the approach enable learning from raw pixels and produce mode-specific behaviors in driving scenarios?

Key findings

• InfoGAIL learns to distinguish and imitate multiple expert modes in a synthetic 2D task where three circle-like trajectories are present.
• In driving experiments, InfoGAIL trained on raw visual inputs can reproduce and separate different driving behaviors corresponding to latent codes with high accuracy (e.g., turning from inner vs outer lane; passing from left vs right).
• Posterior inference Q(c|s,a) achieves over 81% accuracy on latent code identification for pass and near-perfect accuracy for turn in the driving tasks.
• InfoGAIL achieves higher average rollout distances than behavior cloning and standard GAIL, and can even outperform some human demonstrations under reward augmentation and advanced optimization.
• Using transfer learning with pre-trained CNN features (ImageNet) helps enable learning from high-dimensional visual inputs with relatively few demonstrations.