[論文レビュー] Some Considerations on Learning to Explore via Meta-Reinforcement Learning

Bradly C. Stadie, Ge Yang+6 more|arXiv (Cornell University)|Mar 3, 2018

Reinforcement Learning in Robotics参考文献 31被引用数 71

ひとこと要約

この論文は meta-reinforcement learning を、タスクごとのサンプリング分布を迅速に形作ることの学習として再定義し、E-MAML と E-RL² の二つのアルゴリズムを導入し、Krazy World と迷路タスクでその利点を示す。

ABSTRACT

We consider the problem of exploration in meta reinforcement learning. Two new meta reinforcement learning algorithms are suggested: E-MAML and E-$\text{RL}^2$. Results are presented on a novel environment we call `Krazy World' and a set of maze environments. We show E-MAML and E-$\text{RL}^2$ deliver better performance on tasks where exploration is important.

研究の動機と目的

• Interpret meta-RL as learning to quickly find good per-task sampling distributions in new environments.
• Derive gradient-based meta-learning algorithms that optimize exploration during adaptation (E-MAML) and its RL² extension (E-RL²).
• Demonstrate the methods on a high-dimensional Krazy World environment and maze tasks to assess transfer and adaptation speed.

提案手法

• Treat the policy as a sampling distribution over states and optimize how this distribution supports fast adaptation.
• Differentiate the meta-RL objective with respect to the initial sampling distribution to account for its effect on future rewards after adaptation (Eq. 3).
• Derive two-term gradient expression that includes an exploration term affecting the outer meta-update (Eq. 4).
• Define E-MAML as a gradient-based meta-learning variant that explicitly accounts for sampling impact during adaptation.
• Develop E-RL² by modifying the RL² framework to differentiate through sampling via an Explore/Exploit rollout scheme and zeroing rewards from Explore-rollouts during backpropagation.
• Evaluate using Krazy World (high-dimensional, dynamically changing tasks) and maze environments to test sampling differentiation and transfer.

実験結果

リサーチクエスチョン

• RQ1 Can differentiating through the per-task sampling process improve meta-learning adaptation speed and robustness?
• RQ2 Do E-MAML and E-RL² provide faster convergence and better transfer than baseline MAML and RL² across challenging task distributions like Krazy World and mazes?
• RQ3 How does accounting for the initial sampling distribution impact exploration behavior and system identification in meta-RL?
• RQ4 Does the proposed framework reveal superior exploration-driven meta-learning in high-dimensional, dynamically changing environments?

主な発見

• On Krazy World, E-MAML converges faster than MAML, with both achieving good final performance; E-RL² attains the best final performance but with higher initial variance.
• E-RL² generally outperforms baselines in Krazy World by the end of training, while RL² shows high variance and occasional poor performance.
• In maze environments, RL² and E-RL² outperform MAML and E-MAML, benefiting from memory and longer horizon exploration.
• RL² variants tend to solve more mazes over time, indicating memory-based exploration advantages in mazes.
• Overall, the proposed methods yield faster initial gains and improved exploration coverage compared to baselines.