[Paper Review] Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning

Ilya Kostrikov, Kumar Krishna Agrawal+3 more|arXiv (Cornell University)|Sep 9, 2018

Reinforcement Learning in RoboticsComputer Science34 references76 citations

TL;DR

DAC reduces sample complexity and unbiases rewards in adversarial imitation learning by using off-policy discrimination and off-policy actor-critic RL, improving performance with fewer environment interactions.

ABSTRACT

We identify two issues with the family of algorithms based on the Adversarial Imitation Learning framework. The first problem is implicit bias present in the reward functions used in these algorithms. While these biases might work well for some environments, they can also lead to sub-optimal behavior in others. Secondly, even though these algorithms can learn from few expert demonstrations, they require a prohibitively large number of interactions with the environment in order to imitate the expert for many real-world applications. In order to address these issues, we propose a new algorithm called Discriminator-Actor-Critic that uses off-policy Reinforcement Learning to reduce policy-environment interaction sample complexity by an average factor of 10. Furthermore, since our reward function is designed to be unbiased, we can apply our algorithm to many problems without making any task-specific adjustments.

Motivation & Objective

• Identify biases in discriminator-based reward estimation in adversarial imitation learning.
• Reduce policy–environment interaction sample complexity by introducing off-policy components.
• Unbias the learned reward by handling absorbing states explicitly.
• Demonstrate robustness of DAC to noisy, multi-modal, and constrained demonstrations (including human data).
• Show state-of-the-art performance on challenging imitation learning tasks.

Proposed method

• Use an off-policy discriminator trained with a replay buffer to match occupancy measures with the expert.
• Learn absorbing-state rewards to remove biases from terminal states and enable proper handling of episodic tasks.
• Integrate an off-policy RL algorithm (TD3) with the GAIL/AIRL framework to improve sample efficiency.
• Apply gradient penalties to stabilize discriminator training in off-policy settings.
• Use TD3 with rewards provided by the discriminator to train the policy, enabling efficient learning from demonstrations.

Experimental results

Research questions

• RQ1 How do reward biases in discriminator-based imitation learning affect policy performance across tasks?
• RQ2 Can off-policy discrimination and off-policy RL reduce environment interaction requirements while maintaining or improving imitation quality?
• RQ3 Does explicit absorbing-state reward learning improve performance on episodic tasks with survival bonuses or penalties?
• RQ4 Is DAC robust to noisy or sub-optimal human demonstrations and multi-modal expert data?

Key findings

• DAC achieves state-of-the-art imitation learning performance on several continuous control tasks.
• DAC reduces the required environment interactions by an average factor of about 10 compared to prior on-policy AIL methods.
• Using absorbing-state reward learning improves performance in episodic environments and mitigates biases from terminal states.
• DAC remains robust when demonstrations are noisy, multi-modal, or sub-optimal, including human demonstrations.
• Experiments show DAC outperforms GAIL baselines under limited expert demonstrations (e.g., four expert trajectories).
• Reward bias analyses demonstrate that ignoring absorbing-state rewards or using certain discriminator-based rewards can lead to sub-optimal policies, which DAC addresses.