[Paper Review] Generative Adversarial Self-Imitation Learning

Yijie Guo, Junhyuk Oh+2 more|arXiv (Cornell University)|Dec 3, 2018

Reinforcement Learning in Robotics22 references41 citations

TL;DR

GASIL introduces a discriminator-based regularizer that encourages a policy to imitate past good trajectories, treated as internal rewards within a GAIL framework, improving PPO especially with delayed rewards.

ABSTRACT

This paper explores a simple regularizer for reinforcement learning by proposing Generative Adversarial Self-Imitation Learning (GASIL), which encourages the agent to imitate past good trajectories via generative adversarial imitation learning framework. Instead of directly maximizing rewards, GASIL focuses on reproducing past good trajectories, which can potentially make long-term credit assignment easier when rewards are sparse and delayed. GASIL can be easily combined with any policy gradient objective by using GASIL as a learned shaped reward function. Our experimental results show that GASIL improves the performance of proximal policy optimization on 2D Point Mass and MuJoCo environments with delayed reward and stochastic dynamics.

Motivation & Objective

• Motivate the use of an internal learned reward to ease long-term credit assignment in reinforcement learning.
• Develop a regularizer that encourages imitation of past good trajectories within a generative adversarial framework.
• Demonstrate that GASIL can be combined with policy gradient methods to improve performance in challenging environments.
• Show robustness of GASIL to stochastic dynamics and delayed reward signals.

Proposed method

• Define a good-trajectory buffer that stores top-K past trajectories by discounted return.
• Train a discriminator to distinguish between current policy trajectories and good trajectories.
• Update the policy to fool the discriminator by imitating good trajectories, using the GASIL objective.
• Interpret the discriminator as a learned reward function that provides dense feedback along trajectories.
• Optionally combine GASIL with a policy gradient objective by shaping rewards with -log D_phi(s,a).
• Provide algorithmic steps where the discriminator and policy are updated in alternation as in GAN frameworks.

Experimental results

Research questions

• RQ1 Can GASIL improvements be achieved over standard PPO in environments with delayed or sparse rewards?
• RQ2 Is GASIL more sample-efficient than behavior cloning or self-imitation learning in continuous control tasks?
• RQ3 Does GASIL complement policy gradient methods when combined with them?
• RQ4 How robust is GASIL to stochastic dynamics in control tasks?

Key findings

• GASIL improves PPO performance on 2D Point Mass and MuJoCo tasks, especially with delayed rewards.
• GASIL outperforms behavior cloning and self-imitation learning in MuJoCo benchmarks.
• The learned discriminator provides dense rewards that aid exploration and policy improvement.
• GASIL remains robust to some extent under stochastic dynamics, with larger performance gaps under noise in experiments.
• Delayed-reward MuJoCo tasks show larger gains from GASIL compared to non-delayed tasks.