[Paper Review] Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization

Paul Barde, Julien Le Roy+4 more|arXiv (Cornell University)|Jun 23, 2020

Adversarial Robustness in Machine Learning32 references4 citations

TL;DR

This paper proposes Adversarial Soft Advantage Fitting (ASAF), a novel imitation learning framework that eliminates the need for policy optimization by training a structured discriminator conditioned on both the previous policy and a learnable policy. The discriminator's optimization directly yields the optimal generator policy, cutting training complexity in half while matching or outperforming state-of-the-art methods on discrete and continuous control tasks.

ABSTRACT

Adversarial Imitation Learning alternates between learning a discriminator -- which tells apart expert's demonstrations from generated ones -- and a generator's policy to produce trajectories that can fool this discriminator. This alternated optimization is known to be delicate in practice since it compounds unstable adversarial training with brittle and sample-inefficient reinforcement learning. We propose to remove the burden of the policy optimization steps by leveraging a novel discriminator formulation. Specifically, our discriminator is explicitly conditioned on two policies: the one from the previous generator's iteration and a learnable policy. When optimized, this discriminator directly learns the optimal generator's policy. Consequently, our discriminator's update solves the generator's optimization problem for free: learning a policy that imitates the expert does not require an additional optimization loop. This formulation effectively cuts by half the implementation and computational burden of Adversarial Imitation Learning algorithms by removing the Reinforcement Learning phase altogether. We show on a variety of tasks that our simpler approach is competitive to prevalent Imitation Learning methods.

Motivation & Objective

• To simplify adversarial imitation learning by removing the computationally expensive and unstable policy optimization loop.
• To address the instability and sample inefficiency inherent in alternating adversarial training and reinforcement learning in traditional AIL methods.
• To develop a method that simultaneously learns the discriminator and generator policy, eliminating the need for separate RL updates.
• To demonstrate that the proposed method achieves competitive performance with significantly reduced implementation and computational overhead.
• To show that the method generalizes across tasks with varying trajectory lengths, including transition-wise and full-trajectory settings.

Proposed method

• Proposes a structured discriminator that is explicitly conditioned on two policies: the previous generator policy and a learnable policy.
• Uses a maximum entropy formulation where the discriminator estimates the probability of expert vs. generated trajectories using a single parameterized policy.
• Leverages the optimal discriminator form (from Finn et al.) to implicitly solve the generator's optimization problem during discriminator training.
• After training the discriminator for a few epochs, the internal policy model is directly used as the generator policy, bypassing RL entirely.
• Introduces ASAF-w, a variant that processes trajectories in sliding windows of size w, enabling transition-wise imitation learning.
• Employs a shared neural network architecture for both policy and discriminator, with convolutional and fully connected layers followed by MLPs for final prediction.

Experimental results

Research questions

• RQ1 Can adversarial imitation learning be simplified by eliminating the policy optimization loop without sacrificing performance?
• RQ2 Does a discriminator conditioned on both previous and learnable policies naturally recover the optimal generator policy?
• RQ3 Can the proposed method achieve competitive performance on diverse control tasks—both discrete and continuous—without any reinforcement learning updates?
• RQ4 How does the method scale across different trajectory lengths, including full trajectories and transition-wise learning?
• RQ5 Is the method more sample-efficient and stable than standard AIL approaches that rely on alternating RL and discriminator updates?

Key findings

• ASAF achieves performance competitive with state-of-the-art imitation learning methods, including GAIL + PPO, AIRL + PPO, and SQIL, on a range of continuous control tasks such as Hopper, Walker2D, HalfCheetah, and Ant.
• On the MuJoCo environments, ASAF outperforms BC and SQIL and matches or exceeds the performance of AIRL + PPO and GAIL + PPO in terms of mean return.
• In the Pommerman random-tag environment, ASAF-32 achieves a mean return of 1.0 with 150 expert trajectories, demonstrating strong generalization on complex, partial-observability environments.
• The ablation study shows that ASAF-w with window size w=32 achieves better performance than full-trajectory ASAF on some tasks, indicating the benefit of transition-wise learning.
• Hyperparameter search reveals that ASAF is robust to learning rate and window size variations, with the best configurations found via random search across 25 settings.
• The method reduces training complexity by eliminating the reinforcement learning loop, cutting implementation and computational overhead by approximately half compared to standard AIL frameworks.