[Paper Review] Addressing Sample Inefficiency and Reward Bias in Inverse Reinforcement Learning.

Ilya Kostrikov, Kumar Krishna Agrawal+2 more|arXiv (Cornell University)|Sep 9, 2018

Reinforcement Learning in RoboticsComputer Science17 references8 citations

TL;DR

This paper proposes Discriminator-Actor-Critic, a novel IRL algorithm that reduces sample complexity by 10x using off-policy reinforcement learning and eliminates reward function bias through a design that requires no task-specific tuning. The method improves imitation learning efficiency and generalization across diverse environments.

ABSTRACT

We identify two issues with the family of algorithms based on the Adversarial Imitation Learning framework. The first problem is implicit bias present in the reward functions used in these algorithms. While these biases might work well for some environments, they can also lead to sub-optimal behavior in others. Secondly, even though these algorithms can learn from few expert demonstrations, they require a prohibitively large number of interactions with the environment in order to imitate the expert for many real-world applications. In order to address these issues, we propose a new algorithm called Discriminator-Actor-Critic that uses off-policy Reinforcement Learning to reduce policy-environment interaction sample complexity by an average factor of 10. Furthermore, since our reward function is designed to be unbiased, we can apply our algorithm to many problems without making any task-specific adjustments.

Motivation & Objective

• To address the implicit reward bias present in adversarial imitation learning algorithms that can lead to sub-optimal behavior in certain environments.
• To reduce the high sample complexity of IRL algorithms, which currently require prohibitively many environment interactions for real-world deployment.
• To develop a generalizable IRL framework that maintains performance across diverse tasks without requiring task-specific reward function modifications.
• To improve the efficiency and robustness of imitation learning by integrating off-policy RL with a bias-free reward function design.

Proposed method

• The algorithm employs an off-policy reinforcement learning framework to drastically reduce the number of environment interactions required for policy learning.
• It introduces a discriminator-based reward function that is explicitly designed to be unbiased across different environments and tasks.
• The method uses a critic network to estimate the value function, enabling efficient policy updates through off-policy data reuse.
• The actor-critic architecture is trained end-to-end using expert demonstrations, with the discriminator providing reward signals that avoid task-specific engineering.
• The reward function is constructed to minimize implicit biases that commonly affect adversarial imitation learning methods.
• The overall framework enables stable training and generalization across multiple environments without re-tuning the reward function.

Experimental results

Research questions

• RQ1 How can we reduce the sample complexity of inverse reinforcement learning while maintaining high imitation performance?
• RQ2 To what extent can a bias-free reward function improve generalization across diverse environments in IRL?
• RQ3 Can off-policy learning be effectively integrated into IRL to reduce environment interaction requirements?
• RQ4 Does eliminating task-specific reward engineering improve the robustness and transferability of IRL policies?
• RQ5 How does the proposed method compare to existing adversarial IRL approaches in terms of sample efficiency and performance stability?

Key findings

• The proposed algorithm reduces the number of environment interactions required for imitation by an average factor of 10 compared to baseline IRL methods.
• The reward function design successfully eliminates implicit biases, enabling effective performance across a wide range of environments without task-specific adjustments.
• The method achieves stable training and high imitation performance even with limited expert demonstrations.
• Off-policy learning enables efficient reuse of interaction data, significantly improving sample efficiency.
• The algorithm generalizes well to unseen environments due to the unbiased reward function and robust policy training.
• The framework maintains strong performance across diverse tasks without requiring any modifications to the reward function or hyperparameters.