[Paper Review] Learning to reinforcement learn

Jane X. Wang, Zeb Kurth‐Nelson+7 more|arXiv (Cornell University)|Nov 17, 2016

Reinforcement Learning in Robotics32 references379 citations

TL;DR

The paper introduces deep meta-reinforcement learning where a recurrent network trained with one RL algorithm ends up implementing a second, learned RL procedure that adapts rapidly across tasks. It validates the approach through seven proof-of-concept experiments across bandit and MDP settings.

ABSTRACT

In recent years deep reinforcement learning (RL) systems have attained superhuman performance in a number of challenging task domains. However, a major limitation of such applications is their demand for massive amounts of training data. A critical present objective is thus to develop deep RL methods that can adapt rapidly to new tasks. In the present work we introduce a novel approach to this challenge, which we refer to as deep meta-reinforcement learning. Previous work has shown that recurrent networks can support meta-learning in a fully supervised context. We extend this approach to the RL setting. What emerges is a system that is trained using one RL algorithm, but whose recurrent dynamics implement a second, quite separate RL procedure. This second, learned RL algorithm can differ from the original one in arbitrary ways. Importantly, because it is learned, it is configured to exploit structure in the training domain. We unpack these points in a series of seven proof-of-concept experiments, each of which examines a key aspect of deep meta-RL. We consider prospects for extending and scaling up the approach, and also point out some potentially important implications for neuroscience.

Motivation & Objective

• Motivate the need for rapid adaptation in deep RL beyond massive data requirements and task specialization.
• Propose a framework where a recurrent network learns to implement its own RL procedure while being trained with a separate RL algorithm.
• Demonstrate, via proof-of-concept experiments, that the learned RL procedure can exploit task structure and adapt to new tasks.
• Investigate aspects such as adaptive exploration-exploitation, learning-to-learn in bandits, and structure learning in MDPs.
• Discuss potential extensions and plausible neuroscience implications of deep meta-RL.

Proposed method

• Train a recurrent neural network (LSTM) with an RL algorithm to maximize cumulative reward across a distribution of related MDP tasks.
• Feed the network inputs including previous action and received reward to induce internal learning dynamics.
• The recurrent dynamics learn a separate RL procedure that can differ from the outer training algorithm.
• Use Advantage Actor-Critic (A2C/A3C) with entropy regularization and value/policy losses to train the network.
• Evaluate on multiple meta-learning tasks (bandits and MDPs) to assess adaptive learning rates and task-structure exploitation.

Experimental results

Research questions

• RQ1 Can a recurrent network trained with one RL algorithm develop its own internal RL strategy that adapts to new tasks?
• RQ2 Does the learned RL procedure exploit task structure to achieve faster adaptation and lower regret across diverse environments?
• RQ3 To what extent can meta-RL exhibit adaptive exploration and adjustable learning rates in changing environments?
• RQ4 How does meta-RL generalize between related and slightly modified task distributions?
• RQ5 What do these results imply about parallels with model-based/model-free distinctions in neuroscience?

Key findings

• The meta-RL system learns a separate learned RL algorithm within the recurrent dynamics that can differ from the training RL procedure.
• The learned RL procedure can adapt exploration and learning rate to the observed task statistics, improving efficiency.
• Agents trained on structured bandit tasks exploit correlations between arms, performing comparably to Bayesian benchmarks in some settings.
• Training on structured distributions can hurt performance on independent tasks, and vice versa, indicating prior-dependent learning.
• In the two-step task variant, model-free training yielded behavior resembling model-based control under certain analyses, illustrating diverging learned strategies.
• Meta-RL demonstrates adaptive learning rates in volatile environments, outperforming fixed-rate baselines and conventional RL methods.