[Paper Review] Unsupervised Meta-Learning for Reinforcement Learning

Abhishek Gupta, Benjamin Eysenbach+2 more|arXiv (Cornell University)|Jun 12, 2018

Machine Learning and Data ClassificationComputer Science63 references57 citations

TL;DR

The paper introduces unsupervised meta-RL that learns an environment-specific fast-learning procedure by auto-generating tasks via mutual information, then meta-trains with MAML to adapt quickly to new rewards.

ABSTRACT

Meta-learning algorithms use past experience to learn to quickly solve new tasks. In the context of reinforcement learning, meta-learning algorithms acquire reinforcement learning procedures to solve new problems more efficiently by utilizing experience from prior tasks. The performance of meta-learning algorithms depends on the tasks available for meta-training: in the same way that supervised learning generalizes best to test points drawn from the same distribution as the training points, meta-learning methods generalize best to tasks from the same distribution as the meta-training tasks. In effect, meta-reinforcement learning offloads the design burden from algorithm design to task design. If we can automate the process of task design as well, we can devise a meta-learning algorithm that is truly automated. In this work, we take a step in this direction, proposing a family of unsupervised meta-learning algorithms for reinforcement learning. We motivate and describe a general recipe for unsupervised meta-reinforcement learning, and present an instantiation of this approach. Our conceptual and theoretical contributions consist of formulating the unsupervised meta-reinforcement learning problem and describing how task proposals based on mutual information can be used to train optimal meta-learners. Our experimental results indicate that unsupervised meta-reinforcement learning effectively acquires accelerated reinforcement learning procedures without the need for manual task design and these procedures exceed the performance of learning from scratch.

Motivation & Objective

• Reduce human effort in meta-RL by removing manual meta-training task design.
• Enable rapid adaptation to new reward functions within a fixed environment dynamics.
• Show that mutual-information-based task proposals can yield near-oracle meta-learners.
• Demonstrate benefits over learning from scratch and pure exploration followed by fine-tuning.

Proposed method

• Define a CMP (controlled Markov process) with no reward and formulate learning as finding a fast-adapting procedure f.
• Propose task proposals via a parametric reward r_z(s,a) induced by latent z, optimized to minimize worst-case regret.
• Instantiate practical unsupervised meta-learning by using mutual information objectives to generate diverse tasks (DIAYN-based) and a meta-learner (MAML).
• Train a discriminator D_phi to maximize I(z;s) and derive r_z(s,a)=log D_phi(z|s) for task generation.
• Use DIAYN to obtain a latent-conditioned policy and then apply MAML to learn how to learn across the proposed tasks.
• Discuss a random-task baseline (random discriminator) as a comparison.

Experimental results

Research questions

• RQ1 Can unsupervised task proposals eliminate the need for hand-designed meta-training task distributions in meta-RL?
• RQ2 Do mutual-information-based task proposals yield an environment-specific fast-learning procedure that adapts to unseen reward functions?
• RQ3 How does unsupervised meta-RL compare to learning from scratch and to hand-crafted meta-training distributions on benchmark control tasks?

Key findings

• Unsupervised meta-RL accelerates learning compared to learning from scratch across multiple tasks and environments.
• DIAYN-based task proposals generally outperform random task proposals in complex tasks.
• Unsupervised meta-learning can approach the performance of oracle methods that rely on handcrafted task distributions.
• The UML-DIAYN approach often surpasses DIAYN initialization or VIME-based pretraining when fine-tuning on new rewards.
• Results suggest that environment-specific priors learned via unsupervised interaction enhance fast adaptation.