[Paper Review] RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning

Yan Duan, John Schulman+4 more|arXiv (Cornell University)|Nov 9, 2016

Reinforcement Learning in Robotics6 references501 citations

TL;DR

RL2 encodes a fast RL algorithm inside an RNN whose weights are learned by a slow RL outer loop, enabling rapid adaptation to new MDPs and scaling to high-dimensional tasks.

ABSTRACT

Deep reinforcement learning (deep RL) has been successful in learning sophisticated behaviors automatically; however, the learning process requires a huge number of trials. In contrast, animals can learn new tasks in just a few trials, benefiting from their prior knowledge about the world. This paper seeks to bridge this gap. Rather than designing a "fast" reinforcement learning algorithm, we propose to represent it as a recurrent neural network (RNN) and learn it from data. In our proposed method, RL$^2$, the algorithm is encoded in the weights of the RNN, which are learned slowly through a general-purpose ("slow") RL algorithm. The RNN receives all information a typical RL algorithm would receive, including observations, actions, rewards, and termination flags; and it retains its state across episodes in a given Markov Decision Process (MDP). The activations of the RNN store the state of the "fast" RL algorithm on the current (previously unseen) MDP. We evaluate RL$^2$ experimentally on both small-scale and large-scale problems. On the small-scale side, we train it to solve randomly generated multi-arm bandit problems and finite MDPs. After RL$^2$ is trained, its performance on new MDPs is close to human-designed algorithms with optimality guarantees. On the large-scale side, we test RL$^2$ on a vision-based navigation task and show that it scales up to high-dimensional problems.

Motivation & Objective

• Motivate reducing sample complexity by leveraging prior experience through meta-learning for RL.
• Propose RL2, an RNN-based fast RL learner whose inner learning is stored in activations and outer training uses a slow RL algorithm.
• Evaluate RL2 on bandits, tabular MDPs, and vision-based navigation to assess small-scale optimality and large-scale scalability.

Proposed method

• Represent the policy as an RNN (GRU-based) that ingests (s, a, r, d) and outputs actions.
• Treat learning of the fast RL algorithm as an RL problem over a distribution of MDPs, optimizing an objective tied to cumulative discounted reward across trials.
• Train the outer loop with Trust Region Policy Optimization (TRPO) and use a GRU-based baseline to stabilize learning.
• Preserve the RNN state across episodes within a trial to encode the fast-learning dynamics in hidden activations.
• Handle partially observable settings by framing within a POMDP perspective; apply to vision-based tasks (ViZDoom) as proof of scalability.

Experimental results

Research questions

• RQ1 Can RL2 achieve performance close to theoretically optimal algorithms on structured MDP classes like bandits and tabular MDPs?
• RQ2 Does RL2 scale to high-dimensional tasks such as vision-based navigation?
• RQ3 How does RL2 perform relative to established Bayesian and exploration-exploitation methods across varied horizons and state-action spaces?
• RQ4 What are bottlenecks in the outer-loop optimization that limit performance, and can architecture choices mitigate them?

Key findings

• RL2 achieves performance close to theoretically justified algorithms on multi-armed bandits and tabular MDPs across several settings.
• In large-scale vision-based navigation, RL2 demonstrates the ability to utilize visual information and short-term memory accrued over episodes.
• For small horizons in tabular MDPs, RL2 can outperform several baselines, with advantages diminishing as the number of episodes grows.
• In the visual navigation task, RL2 shows substantial reduction in trajectory length from the first to the second episode, indicating effective use of past experience.
• Learning curves reveal variability across random initializations, highlighting sensitivity to outer-loop optimization and architecture.