[論文レビュー] Generative Adversarial Self-Imitation Learning

Yijie Guo, Junhyuk Oh+2 more|arXiv (Cornell University)|Dec 3, 2018

Reinforcement Learning in Robotics参考文献 22被引用数 41

ひとこと要約

GASILは、過去の良い軌跡を模倣するようポリシーを促す識別器ベースの正則化項を導入し、GAILフレームワーク内で内部報酬として扱われることで、遅延報酬がある場合に特にPPOを改善します。

ABSTRACT

This paper explores a simple regularizer for reinforcement learning by proposing Generative Adversarial Self-Imitation Learning (GASIL), which encourages the agent to imitate past good trajectories via generative adversarial imitation learning framework. Instead of directly maximizing rewards, GASIL focuses on reproducing past good trajectories, which can potentially make long-term credit assignment easier when rewards are sparse and delayed. GASIL can be easily combined with any policy gradient objective by using GASIL as a learned shaped reward function. Our experimental results show that GASIL improves the performance of proximal policy optimization on 2D Point Mass and MuJoCo environments with delayed reward and stochastic dynamics.

研究の動機と目的

• Motivate the use of an internal learned reward to ease long-term credit assignment in reinforcement learning.
• Develop a regularizer that encourages imitation of past good trajectories within a generative adversarial framework.
• Demonstrate that GASIL can be combined with policy gradient methods to improve performance in challenging environments.
• Show robustness of GASIL to stochastic dynamics and delayed reward signals.

提案手法

• Define a good-trajectory buffer that stores top-K past trajectories by discounted return.
• Train a discriminator to distinguish between current policy trajectories and good trajectories.
• Update the policy to fool the discriminator by imitating good trajectories, using the GASIL objective.
• Interpret the discriminator as a learned reward function that provides dense feedback along trajectories.
• Optionally combine GASIL with a policy gradient objective by shaping rewards with -log D_phi(s,a).
• Provide algorithmic steps where the discriminator and policy are updated in alternation as in GAN frameworks.

実験結果

リサーチクエスチョン

• RQ1 Can GASIL improvements be achieved over standard PPO in environments with delayed or sparse rewards?
• RQ2 Is GASIL more sample-efficient than behavior cloning or self-imitation learning in continuous control tasks?
• RQ3 Does GASIL complement policy gradient methods when combined with them?
• RQ4 How robust is GASIL to stochastic dynamics in control tasks?

主な発見

• GASIL improves PPO performance on 2D Point Mass and MuJoCo tasks, especially with delayed rewards.
• GASIL outperforms behavior cloning and self-imitation learning in MuJoCo benchmarks.
• The learned discriminator provides dense rewards that aid exploration and policy improvement.
• GASIL remains robust to some extent under stochastic dynamics, with larger performance gaps under noise in experiments.
• Delayed-reward MuJoCo tasks show larger gains from GASIL compared to non-delayed tasks.