[论文解读] Generative Adversarial Self-Imitation Learning

Yijie Guo, Junhyuk Oh+2 more|arXiv (Cornell University)|Dec 3, 2018

Reinforcement Learning in Robotics参考文献 22被引用 41

一句话总结

GASIL 引入基于判别器的正则化项，促使策略模仿过去的良好轨迹，作为在 GAIL 框架内的内部奖励，从而在延迟奖励的情况下提升 PPO。

ABSTRACT

This paper explores a simple regularizer for reinforcement learning by proposing Generative Adversarial Self-Imitation Learning (GASIL), which encourages the agent to imitate past good trajectories via generative adversarial imitation learning framework. Instead of directly maximizing rewards, GASIL focuses on reproducing past good trajectories, which can potentially make long-term credit assignment easier when rewards are sparse and delayed. GASIL can be easily combined with any policy gradient objective by using GASIL as a learned shaped reward function. Our experimental results show that GASIL improves the performance of proximal policy optimization on 2D Point Mass and MuJoCo environments with delayed reward and stochastic dynamics.

研究动机与目标

• Motivate the use of an internal learned reward to ease long-term credit assignment in reinforcement learning.
• Develop a regularizer that encourages imitation of past good trajectories within a generative adversarial framework.
• Demonstrate that GASIL can be combined with policy gradient methods to improve performance in challenging environments.
• Show robustness of GASIL to stochastic dynamics and delayed reward signals.

提出的方法

• Define a good-trajectory buffer that stores top-K past trajectories by discounted return.
• Train a discriminator to distinguish between current policy trajectories and good trajectories.
• Update the policy to fool the discriminator by imitating good trajectories, using the GASIL objective.
• Interpret the discriminator as a learned reward function that provides dense feedback along trajectories.
• Optionally combine GASIL with a policy gradient objective by shaping rewards with -log D_phi(s,a).
• Provide algorithmic steps where the discriminator and policy are updated in alternation as in GAN frameworks.

实验结果

研究问题

• RQ1 Can GASIL improvements be achieved over standard PPO in environments with delayed or sparse rewards?
• RQ2 Is GASIL more sample-efficient than behavior cloning or self-imitation learning in continuous control tasks?
• RQ3 Does GASIL complement policy gradient methods when combined with them?
• RQ4 How robust is GASIL to stochastic dynamics in control tasks?

主要发现

• GASIL improves PPO performance on 2D Point Mass and MuJoCo tasks, especially with delayed rewards.
• GASIL outperforms behavior cloning and self-imitation learning in MuJoCo benchmarks.
• The learned discriminator provides dense rewards that aid exploration and policy improvement.
• GASIL remains robust to some extent under stochastic dynamics, with larger performance gaps under noise in experiments.
• Delayed-reward MuJoCo tasks show larger gains from GASIL compared to non-delayed tasks.