[Paper Review] A Pragmatic Look at Deep Imitation Learning

Kai Arulkumaran, Dan Ogawa Lillrank|arXiv (Cornell University)|Aug 4, 2021

Reinforcement Learning in Robotics45 references4 citations

TL;DR

This paper conducts a fair, empirical comparison of six deep imitation learning algorithms—GAIL, AdRIL, BC, and updated off-policy variants of GMMIL, DRIL, and RED—using a common SAC-based off-policy framework and D4RL MuJoCo benchmarks. GAIL consistently performs best across trajectory budgets, AdRIL offers strong performance with simpler tuning, and behavioral cloning remains highly competitive with sufficient data.

ABSTRACT

The introduction of the generative adversarial imitation learning (GAIL) algorithm has spurred the development of scalable imitation learning approaches using deep neural networks. Many of the algorithms that followed used a similar procedure, combining on-policy actor-critic algorithms with inverse reinforcement learning. More recently there have been an even larger breadth of approaches, most of which use off-policy algorithms. However, with the breadth of algorithms, everything from datasets to base reinforcement learning algorithms to evaluation settings can vary, making it difficult to fairly compare them. In this work we re-implement 6 different IL algorithms, updating 3 of them to be off-policy, base them on a common off-policy algorithm (SAC), and evaluate them on a widely-used expert trajectory dataset (D4RL) for the most common benchmark (MuJoCo). After giving all algorithms the same hyperparameter optimisation budget, we compare their results for a range of expert trajectories. In summary, GAIL, with all of its improvements, consistently performs well across a range of sample sizes, AdRIL is a simple contender that performs well with one important hyperparameter to tune, and behavioural cloning remains a strong baseline when data is more plentiful.

Motivation & Objective

• To address the lack of fair, reproducible comparisons in deep imitation learning due to inconsistent evaluation protocols, datasets, and algorithmic choices.
• To re-implement and update six IL algorithms (including GAIL, AdRIL, and others) to use off-policy reinforcement learning (SAC) for consistent training and evaluation.
• To evaluate all methods under the same hyperparameter optimization budget and on the same expert trajectory dataset (D4RL) across multiple MuJoCo environments.
• To identify which IL methods remain robust and practical under realistic data constraints and implementation variations.
• To release a unified, open-source codebase to enable future fair benchmarking and reproducibility in imitation learning research.

Proposed method

• Re-implemented six deep IL algorithms—GAIL, GMMIL, RED, DRIL, AdRIL, and PWIL—using soft actor-critic (SAC) as the base off-policy reinforcement learning algorithm.
• Updated three on-policy IL methods (GMMIL, RED, DRIL) to operate in off-policy settings, enabling consistent comparison with SAC-based training.
• Used the D4RL dataset of expert trajectories from MuJoCo environments as the standard benchmark for evaluation.
• Applied a shared hyperparameter optimization budget across all algorithms to ensure fair comparison, using Bayesian optimization with 50 trials per method.
• Reported results using current best practices: multiple random seeds, mean and standard deviation, and statistical significance assessment.
• Diagnosed training failures in off-policy DRIL, GMMIL, and RED by monitoring reward trends and Q-values, and explored online discriminator training variants.

Experimental results

Research questions

• RQ1 How do different deep imitation learning algorithms perform when trained under identical conditions using a common off-policy RL backbone (SAC) and the same expert dataset?
• RQ2 Does the performance of on-policy IL algorithms degrade when converted to off-policy settings, and if so, why?
• RQ3 How do hyperparameter choices (e.g., batch size, discriminator frequency) scale with trajectory budget size across different IL methods?
• RQ4 Can simpler IL methods like AdRIL match or exceed the performance of more complex ones like GAIL, especially with limited expert data?
• RQ5 What role does behavioral cloning (BC) play as a baseline when expert data is abundant, and how does its performance scale with data quantity?

Key findings

• GAIL consistently achieved the highest performance across all trajectory budgets, with a mean return of 11,392.09 ± 377.15 on the HalfCheetah environment with 25 trajectories.
• AdRIL performed comparably to GAIL at higher trajectory budgets (25 trajectories), achieving 5,055.76 ± 64.13 return, and required tuning only one key hyperparameter: discriminator update frequency.
• Behavioral cloning (BC) became a strong baseline with sufficient data, outperforming several adversarial IL methods when 25 expert trajectories were available.
• The off-policy versions of DRIL, GMMIL, and RED failed to converge despite extensive hyperparameter tuning and regularization, suggesting fundamental incompatibilities when transitioning from on-policy to off-policy training.
• Hyperparameter tuning revealed that batch size and discriminator size generally increased with trajectory budget, though this trend was not consistent (e.g., optimal batch size for GAIL decreased with more data).
• The original on-policy versions of DRIL, GMMIL, and RED were successfully optimized, indicating that the failure was specifically due to the off-policy adaptation, not the algorithms themselves.