[Paper Review] What Matters for Adversarial Imitation Learning?

Manu Orsini, Anton Raichuk+8 more|arXiv (Cornell University)|Jun 1, 2021

Reinforcement Learning in RoboticsComputer Science49 references18 citations

TL;DR

This paper conducts a large-scale empirical study of over 500,000 adversarial imitation learning (AIL) agents across 10 continuous-control tasks to evaluate the impact of algorithmic choices and demonstration types. It reveals that human demonstrations are fundamentally different from synthetic ones, and that common practices—like using specific discriminator regularizers—may not generalize, challenging assumptions in prior AIL research.

ABSTRACT

Adversarial imitation learning has become a popular framework for imitation in continuous control. Over the years, several variations of its components were proposed to enhance the performance of the learned policies as well as the sample complexity of the algorithm. In practice, these choices are rarely tested all together in rigorous empirical studies. It is therefore difficult to discuss and understand what choices, among the high-level algorithmic options as well as low-level implementation details, matter. To tackle this issue, we implement more than 50 of these choices in a generic adversarial imitation learning framework and investigate their impacts in a large-scale study (>500k trained agents) with both synthetic and human-generated demonstrations. While many of our findings confirm common practices, some of them are surprising or even contradict prior work. In particular, our results suggest that artificial demonstrations are not a good proxy for human data and that the very common practice of evaluating imitation algorithms only with synthetic demonstrations may lead to algorithms which perform poorly in the more realistic scenarios with human demonstrations.

Motivation & Objective

• To systematically evaluate the impact of over 50 hyperparameter and architectural choices in adversarial imitation learning (AIL) in a controlled, large-scale setting.
• To investigate whether synthetic demonstrations are a valid proxy for human demonstrations in AIL evaluation.
• To identify which algorithmic components (e.g., RL algorithm, discriminator regularizer, training ratio) most significantly affect policy performance and sample efficiency.
• To provide actionable, data-driven recommendations for designing and evaluating new AIL algorithms in realistic settings.
• To expose the limitations of prior work that evaluates only on synthetic demonstrations, which may mislead algorithm design and performance assessment.

Proposed method

• Implemented a highly configurable generic AIL framework supporting 4 RL algorithms, 7 discriminator regularizers, and over 50 hyperparameters.
• Conducted a large-scale experimental study with more than 500,000 trained agents across 10 continuous-control environments (OpenAI Gym and Adroit).
• Used both synthetic demonstrations (from trained RL agents) and human-generated demonstrations to evaluate generalization across data sources.
• Employed standardized performance metrics: final episode return and average return during training, with quantile analysis across configurations.
• Performed ablation studies on key hyperparameters, including batch size, replay ratio, discriminator-to-RL update ratio, and number of combined batches.
• Analyzed results using conditional performance quantiles and distributional analysis of top-performing configurations to isolate influential factors.

Experimental results

Research questions

• RQ1 Which hyperparameter choices in AIL have the most significant impact on final policy performance and sample efficiency?
• RQ2 How do different discriminator regularizers (e.g., gradient penalty, weight decay, dropout) compare in performance across environments and demonstration types?
• RQ3 To what extent do synthetic demonstrations accurately reflect the behavior and challenges of human-generated demonstrations in AIL?
• RQ4 How do the relative performances of algorithmic choices (e.g., on-policy vs. off-policy, reward shaping) vary between synthetic and human demonstrations?
• RQ5 What is the optimal configuration of training hyperparameters (e.g., batch size, replay ratio, update ratio) for maximizing imitation performance across diverse tasks?

Key findings

• Standard discriminator regularizers like gradient penalty do not consistently outperform simpler, standard supervised learning techniques such as weight decay or dropout, especially in easier environments.
• For easier environments, explicit discriminator regularization is often unnecessary—policies trained without any regularizer can achieve strong performance.
• Human demonstrations lead to significantly lower performance across all AIL configurations compared to synthetic demonstrations, indicating a fundamental distribution shift.
• The relative performance of algorithmic choices (e.g., choice of RL algorithm, regularizer) depends heavily on the source of demonstrations—results on synthetic data do not generalize to human data.
• The best-performing configurations for synthetic demonstrations often fail on human demonstrations, suggesting that evaluating only on synthetic data leads to misleading conclusions about algorithm robustness.
• Hyperparameters like batch size, replay ratio, and discriminator-to-RL update ratio show strong, non-monotonic effects on performance, with optimal values varying by environment and demonstration type.