[Paper Review] Self-Imitation Learning

Junhyuk Oh, Yijie Guo+2 more|arXiv (Cornell University)|Jun 14, 2018

Innovative Teaching and Learning MethodsPsychology69 citations

TL;DR

Self-Imitation Learning (SIL) is an off-policy actor-critic method that imitates the agent’s past good decisions to drive deep exploration, improving performance on hard exploration Atari games and MuJoCo tasks.

ABSTRACT

This paper proposes Self-Imitation Learning (SIL), a simple off-policy actor-critic algorithm that learns to reproduce the agent's past good decisions. This algorithm is designed to verify our hypothesis that exploiting past good experiences can indirectly drive deep exploration. Our empirical results show that SIL significantly improves advantage actor-critic (A2C) on several hard exploration Atari games and is competitive to the state-of-the-art count-based exploration methods. We also show that SIL improves proximal policy optimization (PPO) on MuJoCo tasks.

Motivation & Objective

• Investigate whether exploiting past good experiences can improve reinforcement learning performance.
• Develop a simple off-policy objective for actor-critic architectures that leverages past successes.
• Theoretically justify SIL via a lower-bound soft Q-learning framework.
• Empirically evaluate SIL on hard exploration Atari games and continuous control benchmarks.
• Assess whether SIL complements existing exploration methods and other algorithms like PPO.

Proposed method

• Store past episode trajectories with cumulative rewards in a replay buffer.
• Use an off-policy SIL objective that updates when R > V_theta(s) with two loss components: L_sil_policy and L_sil_value.
• Sample from the replay buffer with prioritized replay using (R-V_theta(s))_+ as the priority.
• Combine SIL with on-policy A2C updates to form A2C+SIL.
• Extend SIL to PPO and test across Atari and MuJoCo tasks.
• Provide theoretical justification linking SIL to lower-bound-soft-Q-learning within entropy-regularized RL.

Experimental results

Research questions

• RQ1 Does self-imitation learning improve exploration and overall performance in RL agents?
• RQ2 Can SIL complement count-based exploration methods on hard exploration tasks?
• RQ3 Is SIL broadly applicable to both discrete and continuous control and compatible with PPO?
• RQ4 Under what conditions does exploiting past experiences help or hinder learning?

Key findings

• SIL improves A2C on several hard exploration Atari games and is competitive with count-based exploration methods.
• SIL also improves PPO on MuJoCo continuous-control tasks.
• Across 49 Atari games, A2C+SIL outperforms A2C in median human-normalized score and in most hard exploration games.
• SIL is complementary to exploration methods and can drive deep exploration by exploiting rare positive experiences.
• A2C+SIL can outperform state-of-the-art count-based methods on multiple hard exploration tasks.
• In some cases, early-stage exploitation may hinder performance, suggesting a need for adaptive SIL usage.