[Paper Review] Addressing reward bias in Adversarial Imitation Learning with neutral reward functions

Rohit Jena, Siddharth Agrawal+1 more|arXiv (Cornell University)|Sep 20, 2020

Adversarial Robustness in Machine Learning14 references4 citations

TL;DR

This paper proposes a neutral reward function for Generative Adversarial Imitation Learning (GAIL) that eliminates both survival and termination bias in task-based environments with single or multiple terminal states. By replacing standard positive or negative reward functions with a symmetric log-ratio form, the method achieves superior imitation performance, outperforming existing GAIL and DAC approaches on environments with complex termination dynamics.

ABSTRACT

Generative Adversarial Imitation Learning suffers from the fundamental problem of reward bias stemming from the choice of reward functions used in the algorithm. Different types of biases also affect different types of environments - which are broadly divided into survival and task-based environments. We provide a theoretical sketch of why existing reward functions would fail in imitation learning scenarios in task based environments with multiple terminal states. We also propose a new reward function for GAIL which outperforms existing GAIL methods on task based environments with single and multiple terminal states and effectively overcomes both survival and termination bias.

Motivation & Objective

• To identify and address reward bias in GAIL, particularly survival and termination bias, which hinder imitation in task-based environments.
• To analyze how existing reward functions (positive/negative) lead to suboptimal policies in environments with multiple terminal states.
• To propose a neutral reward function that is unbiased toward survival or early termination, enabling robust imitation across diverse task-based settings.
• To empirically validate the superiority of the neutral reward function over existing methods like DAC and standard GAIL variants in environments with single and multiple terminal states.

Proposed method

• Proposes a new reward function defined as $ R = \log(D(s,a)) - \log(1 - D(s,a)) $, where $ D(s,a) $ is the discriminator's output, to create a symmetric, neutral reward signal.
• Uses this neutral reward in the GAIL framework, replacing the standard $ \log(D(s,a)) $ or $ -\log(1 - D(s,a)) $ forms to avoid bias toward survival or termination.
• Employs a standard GAIL training pipeline with an on-policy RL algorithm, where the policy is optimized to maximize the neutral reward signal from the discriminator.
• Trains the discriminator to distinguish expert transitions from expert and expert-imitating trajectories, maintaining the GAIL objective but with a balanced reward signal.
• Uses a 2-layer MLP with Leaky ReLU activations for both policy and value networks, and averages results across 3 random seeds for robustness.
• Evaluates performance using success rate and environment return across 5 MiniGrid environments with varying terminal state structures.

Experimental results

Research questions

• RQ1 How do positive and negative reward functions in GAIL induce survival and termination bias in task-based environments with multiple terminal states?
• RQ2 Why do existing methods like DAC fail in task-based environments with multiple terminal states, despite addressing bias in simpler settings?
• RQ3 Can a neutral reward function that avoids bias toward survival or termination improve imitation performance in complex task-based environments?
• RQ4 Does the proposed neutral reward function outperform both positive and negative reward baselines and DAC in environments with single and multiple terminal states?

Key findings

• The neutral reward function achieved a success rate of 1.00 ± 0.00 on the Empty and DoorKey environments, matching expert performance and outperforming all baselines.
• On GoToDoor, the neutral reward achieved 0.97 ± 0.15 success rate, significantly outperforming DAC (0.26 ± 0.44) and negative reward (0.38 ± 0.48).
• In the RedBlueDoors environment, the neutral reward reached 0.91 ± 0.27 success rate, surpassing DAC (0.70 ± 0.45) and negative reward (0.00 ± 0.00).
• In DistShift, the neutral reward achieved 0.97 ± 0.15 success rate, outperforming DAC (0.76 ± 0.42) and negative reward (0.65 ± 0.47).
• GAIL with positive rewards failed entirely in environments with multiple terminal states, achieving 0.00 ± 0.00 success rate across all such environments.
• The negative reward function achieved only 0.25 ± 0.43 success rate in GoToDoor, indicating strong termination bias due to random chance of reaching the correct door.