[Paper Review] Meta-Reinforcement Learning of Structured Exploration Strategies

Abhishek Gupta, Russell Mendonca+3 more|arXiv (Cornell University)|Feb 20, 2018

Reinforcement Learning in Robotics30 references180 citations

TL;DR

Introduces MAESN, a gradient-based meta-RL method that learns structured, temporally coherent exploration via a learned latent space, enabling fast adaptation to new tasks. Demonstrates improved exploration and adaptation over prior meta-RL methods and task-agnostic exploration across robotics and locomotion tasks.

ABSTRACT

Exploration is a fundamental challenge in reinforcement learning (RL). Many of the current exploration methods for deep RL use task-agnostic objectives, such as information gain or bonuses based on state visitation. However, many practical applications of RL involve learning more than a single task, and prior tasks can be used to inform how exploration should be performed in new tasks. In this work, we explore how prior tasks can inform an agent about how to explore effectively in new situations. We introduce a novel gradient-based fast adaptation algorithm -- model agnostic exploration with structured noise (MAESN) -- to learn exploration strategies from prior experience. The prior experience is used both to initialize a policy and to acquire a latent exploration space that can inject structured stochasticity into a policy, producing exploration strategies that are informed by prior knowledge and are more effective than random action-space noise. We show that MAESN is more effective at learning exploration strategies when compared to prior meta-RL methods, RL without learned exploration strategies, and task-agnostic exploration methods. We evaluate our method on a variety of simulated tasks: locomotion with a wheeled robot, locomotion with a quadrupedal walker, and object manipulation.

Motivation & Objective

• Motivate improved exploration by leveraging multiple related prior tasks in meta-RL.
• Develop a gradient-based adaptation method that injects structured stochasticity into exploration.
• Enable fast adaptation to new tasks through policy gradient updates and a learned latent exploration space.
• Show that structured noise leads to more coherent exploration than random action-space noise.
• Compare MAESN against prior meta-RL methods and task-agnostic exploration in diverse robotics domains.

Proposed method

• Propose MAESN, combining structured latent-space noise with MAML-style gradient adaptation.
• Represent exploration as a policy conditioned on per-episode latent variables z drawn from a learnable Gaussian, enabling temporally coherent exploration.
• Meta-train policy parameters and per-task latent parameters to maximize post-update rewards, with KL regularization to a unit Gaussian prior.
• Inner loop: update latent parameters and optionally policy parameters via policy gradient; outer loop: meta-update using TRPO.
• Use reparameterization or likelihood-ratio methods to backpropagate through z sampling during adaptation.
• Evaluate by comparing against MAML, RL2, latent-space baselines, and training-from-scratch on manipulation and locomotion tasks.

Experimental results

Research questions

• RQ1 Can meta-learned, structured exploration strategies enable coherent exploration and rapid adaptation to new sparse-reward tasks?
• RQ2 Does MAESN outperform prior meta-RL methods (e.g., MAML, RL2) and latent-space baselines in learning to explore?
• RQ3 What kinds of exploration strategies do MAESN-trained policies exhibit, and how do they compare visually to baselines?
• RQ4 Which components of MAESN (latent space, meta-training objective, gradient updates) are critical for performance?

Key findings

• MAESN learns temporally coherent exploration by conditioning actions on a per-episode latent variable, improving exploration over baselines.
• MAESN adapts quickly to novel tasks with sparse rewards, outperforming MAML and RL2 in several domains.
• Latent-space meta-training enables effective adaptation, outperforming latent-space-only approaches.
• Exploration trajectories with MAESN cover coherent behaviors that better match task distributions than random exploration or MAML.
• The latent parameters pre-update converge toward the prior, while post-update parameters move to task-specific regions, enabling effective adaptation.
• Structured noise is essential for exploring diverse, task-relevant strategies beyond time-invariant action noise.