[Paper Review] Online and Offline Reinforcement Learning by Planning with a Learned Model

Julian Schrittwieser, Thomas Hubert+4 more|arXiv (Cornell University)|Apr 13, 2021

Reinforcement Learning in Robotics31 references26 citations

TL;DR

The paper introduces Reanalyse, a model-based planning technique that improves training targets using a learned model, enabling MuZero to operate effectively across online, offline, and demonstration-driven data regimes, culminating in MuZero Unplugged which achieves state-of-the-art results in both online Atari and offline RL benchmarks.

ABSTRACT

Learning efficiently from small amounts of data has long been the focus of model-based reinforcement learning, both for the online case when interacting with the environment and the offline case when learning from a fixed dataset. However, to date no single unified algorithm could demonstrate state-of-the-art results in both settings. In this work, we describe the Reanalyse algorithm which uses model-based policy and value improvement operators to compute new improved training targets on existing data points, allowing efficient learning for data budgets varying by several orders of magnitude. We further show that Reanalyse can also be used to learn entirely from demonstrations without any environment interactions, as in the case of offline Reinforcement Learning (offline RL). Combining Reanalyse with the MuZero algorithm, we introduce MuZero Unplugged, a single unified algorithm for any data budget, including offline RL. In contrast to previous work, our algorithm does not require any special adaptations for the off-policy or offline RL settings. MuZero Unplugged sets new state-of-the-art results in the RL Unplugged offline RL benchmark as well as in the online RL benchmark of Atari in the standard 200 million frame setting.

Motivation & Objective

• Improve data efficiency in online RL by reusing stored data to generate better training targets.
• enable fully offline RL by learning from datasets without environment interaction.
• Demonstrate that a single algorithm (MuZero Unplugged) can handle online, offline, and demonstration-based learning across discrete and continuous action spaces.
• Show scalability to high-dimensional state spaces (pixel inputs) and various data budgets.

Proposed method

• Use model-based policy and value improvement operators to generate new training targets from stored data.
• Integrate Reanalyse with MuZero's MCTS planning and learned world model to produce updated targets for p (policy), v (value), and r (reward).
• Unroll the learned model for K steps and train on multi-step TD targets drawn from reanalysed trajectories.
• Adjust training targets with the tree search's improved policy pi_t and value z_t to create a virtuous cycle of model and target improvements.
• Define Reanalyse fraction as the ratio of targets derived from environment interaction vs. reanalysed data to control the data budget.

Experimental results

Research questions

• RQ1 Can a single algorithm achieve strong performance in both online RL and offline RL settings without special off-policy adaptations?
• RQ2 How does reusing learned models to reanalyse past data affect learning efficiency across different data budgets?
• RQ3 Does combining Reanalyse with MuZero enable effective learning from demonstrations and purely offline datasets?
• RQ4 How does MuZero Unplugged perform on pixel-based and continuous-action tasks compared to specialized offline RL baselines?
• RQ5 What data-budget regimes (online, offline, mixed) are feasible with a unified planning-based approach?

Key findings

• Reanalyse enables data-efficient learning across orders of magnitude of data budgets by reusing stored data for improved targets.
• MuZero Unplugged achieves state-of-the-art results on RL Unplugged (offline Atari and continuous control) and online Atari benchmarks without special offline adaptations.
• In Atari, MuZero Unplugged outperformed baselines across 44 of 46 games when using offline data or mixed budgets.
• MuZero Unplugged also matches or exceeds baselines on the DM Control Suite for continuous control tasks.
• A combination of MCTS-based action selection and Reanalyse loss yields the best performance, establishing MuZero Unplugged as a robust unified RL method.