[Paper Review] Meta-Gradient Reinforcement Learning with an Objective Discovered Online

Zhongwen Xu, Hado van Hasselt+4 more|arXiv (Cornell University)|Jul 16, 2020

Reinforcement Learning in Robotics39 references35 citations

TL;DR

The paper introduces FRODO, an online meta-gradient framework that discovers and updates the RL objective itself, enabling online adaptation and improved learning efficiency across tasks, including Atari games.

ABSTRACT

Deep reinforcement learning includes a broad family of algorithms that parameterise an internal representation, such as a value function or policy, by a deep neural network. Each algorithm optimises its parameters with respect to an objective, such as Q-learning or policy gradient, that defines its semantics. In this work, we propose an algorithm based on meta-gradient descent that discovers its own objective, flexibly parameterised by a deep neural network, solely from interactive experience with its environment. Over time, this allows the agent to learn how to learn increasingly effectively. Furthermore, because the objective is discovered online, it can adapt to changes over time. We demonstrate that the algorithm discovers how to address several important issues in RL, such as bootstrapping, non-stationarity, and off-policy learning. On the Atari Learning Environment, the meta-gradient algorithm adapts over time to learn with greater efficiency, eventually outperforming the median score of a strong actor-critic baseline.

Motivation & Objective

• Motivate learning an RL objective instead of handcrafting it, using online meta-gradient descent.
• Develop a meta-network that parameterises the RL update target to be learned from experience.
• Enable online adaptation of the objective to handle bootstrapping, non-stationarity, and off-policy learning.
• Demonstrate that online objective discovery can outperform fixed baselines on complex environments.

Proposed method

• Parameterise the RL update target G with a meta-network g_eta that maps trajectories to a scalar target.
• Apply meta-gradient descent to optimise eta by backpropagating through M inner updates to the agent parameters theta.
• Use an outer loss L_outer on a validation trajectory to compute meta-gradients for eta.
• Instantiate the framework as FRODO (Flexible Reinforcement Objective Discovered Online) and apply to prediction, value-based control, and actor-critic settings.
• In large-scale experiments, incorporate a consistency loss to regularise G_eta toward time-self-consistency, and employ VTrace for off-policy correction.

Experimental results

Research questions

• RQ1 Can an online meta-gradient approach discover useful RL update targets without hand-crafted objectives?
• RQ2 How can online objective discovery address bootstrapping and non-stationarity in RL?
• RQ3 Does online discovery of the update target yield improvements over strong actor-critic baselines on diverse tasks?
• RQ4 What is the effect of representing the objective as a learned target versus a learned loss in meta-gradients?

Key findings

• In toy domains, the learned target enables bootstrapping and non-stationarity handling, outperforming fixed look-ahead baselines.
• On 57 Atari games, the meta-gradient algorithm outperforms the median score of a strong actor-critic baseline after sufficient training.
• The learned target diverges from VTrace targets across games, indicating the discovery of distinct online objectives.
• Representing the target (not the loss) yields more stable online learning than representing a loss.
• A modest consistency loss (regularisation towards self-consistency) significantly speeds up learning and improves performance, while too strong regularisation harms results.