[Paper Review] Generative Adversarial Imitation Learning

Jonathan Ho, Stefano Ermon|arXiv (Cornell University)|Jun 10, 2016

Reinforcement Learning in Robotics21 references369 citations

TL;DR

Introduces a model-free imitation learning algorithm inspired by generative adversarial networks to directly learn policies from expert demonstrations without reinforcement signals or expert interaction. It frames imitation as occupancy-measure matching via a GAN-like setup and achieves strong performance on high-dimensional control tasks.

ABSTRACT

Consider learning a policy from example expert behavior, without interaction with the expert or access to reinforcement signal. One approach is to recover the expert's cost function with inverse reinforcement learning, then extract a policy from that cost function with reinforcement learning. This approach is indirect and can be slow. We propose a new general framework for directly extracting a policy from data, as if it were obtained by reinforcement learning following inverse reinforcement learning. We show that a certain instantiation of our framework draws an analogy between imitation learning and generative adversarial networks, from which we derive a model-free imitation learning algorithm that obtains significant performance gains over existing model-free methods in imitating complex behaviors in large, high-dimensional environments.

Motivation & Objective

• Motivate direct policy learning from expert trajectory data without querying the expert or using a reinforcement signal.
• Develop a scalable imitation learning framework that bypasses inverse reinforcement learning as an intermediate step.
• Leverage adversarial training to align the learner's state-action occupancy with the expert's occupancy.
• Demonstrate performance gains over existing model-free imitation methods on high-dimensional environments.

Proposed method

• Formulate imitation as minimizing the Jensen-Shannon divergence between occupancy measures plus a causal entropy regularizer.
• Introduce a regularizer psi_GA whose convex conjugate induces a soft occupancy-measure matching objective.
• Derive a practical algorithm (Generative Adversarial Imitation Learning) that alternates between training a discriminator D to distinguish learner vs. expert transitions and updating the policy π via a TRPO-like step using log(D) as the cost.
• Connect the algorithm to GANs by treating the learner's occupancy as the generator distribution and the expert's occupancy as the target distribution.
• Use neural network parameterizations for both the policy and the discriminator and apply an Adam update for D and a KL-constrained natural gradient step for the policy.

Experimental results

Research questions

• RQ1 Can direct policy learning from expert trajectories be achieved without IRL or reinforcement signals?
• RQ2 Is there a model-free imitation method that scales to high-dimensional, continuous control tasks while beating existing baselines?
• RQ3 Does an adversarial training objective yield closer occupancy matching to the expert than traditional apprenticeship learning methods?

Key findings

• The proposed Generative Adversarial Imitation Learning (GAIL) outperforms behavioral cloning, FEM, and GTAL on several physics-based control tasks.
• GAIL achieves near-expert performance across a range of dataset sizes in classic control tasks like cartpole, acrobot, and mountain car.
• On several MuJoCo environments, GAIL provides a large performance boost over baselines, often reaching a substantial fraction of expert performance.
• In some tasks (e.g., Reacher) causal entropy regularization can yield additional improvements in sample efficiency.
• GAIL directly minimizes a true distance between occupancy measures (JS divergence) rather than relying on linear or convex cost-function classes, enabling closer imitation to the expert.