[Paper Review] Discovery of Useful Questions as Auxiliary Tasks

Vivek Veeriah, Matteo Hessel+7 more|arXiv (Cornell University)|Sep 10, 2019

Neural Networks and Reservoir ComputingComputer Science38 citations

TL;DR

The paper introduces a multi-step meta-gradient method to discover GVF-based questions that serve as auxiliary tasks, enabling learned representations to support the main RL task and improve data efficiency, including Atari benchmarks.

ABSTRACT

Arguably, intelligent agents ought to be able to discover their own questions so that in learning answers for them they learn unanticipated useful knowledge and skills; this departs from the focus in much of machine learning on agents learning answers to externally defined questions. We present a novel method for a reinforcement learning (RL) agent to discover questions formulated as general value functions or GVFs, a fairly rich form of knowledge representation. Specifically, our method uses non-myopic meta-gradients to learn GVF-questions such that learning answers to them, as an auxiliary task, induces useful representations for the main task faced by the RL agent. We demonstrate that auxiliary tasks based on the discovered GVFs are sufficient, on their own, to build representations that support main task learning, and that they do so better than popular hand-designed auxiliary tasks from the literature. Furthermore, we show, in the context of Atari 2600 videogames, how such auxiliary tasks, meta-learned alongside the main task, can improve the data efficiency of an actor-critic agent.

Motivation & Objective

• Motivate agents to autonomously discover useful questions that can be answered to acquire valuable representations for the main RL task.
• Propose a principled method to automate GVF-question discovery using meta-gradients to optimize representation usefulness for the main task.
• Demonstrate that GVF-based auxiliary tasks discovered this way can be sufficient for representation learning and can outperform hand-designed auxiliary tasks.
• Show that meta-learned GVFs can improve data efficiency in Atari when learned alongside the main task.
• Assess the benefits and limitations of non-myopic (multi-step) meta-gradients for discovery across domains.

Proposed method

• Present a neural architecture with a main task network (policy/value) and a separate question network that parameterizes GVF cumulants and discounts.
• Use non-myopic multi-step meta-gradients to update meta-parameters of the question network by backpropagating through L inner RL updates to a meta-loss.
• Instantiate the approach in an actor-critic setting where GVF answers are learned via generalized TD updates along the on-policy main task.
• Compare discovered GVFs against handcrafted baselines (reward prediction, pixel control, random GVFs) in gridworld, Collect-Objects, and Atari domains.
• Evaluate representation-learning scenarios (GVFs alone suffice for main task learning) and joint learning scenarios (GVFs plus main task updates for data efficiency).

Experimental results

Research questions

• RQ1 Can meta-gradients discover GVF-questions such that learning answers to them suffices to learn representations for complex RL tasks?
• RQ2 Do discovered GVF-based auxiliary tasks improve data efficiency when learned alongside the main RL task, compared to hand-designed auxiliary tasks?
• RQ3 How do the number of GVF questions and the meta-unroll length affect learning stability and performance?
• RQ4 Are non-myopic meta-gradients essential for discovering useful GVFs in large domains?
• RQ5 How do discovered GVFs perform relative to handcrafted auxiliary tasks across multiple domains including Atari?

Key findings

• Discovered GVFs can yield representations that support optimal policies in gridworlds and competitive performance on Atari after training.
• GVF-based auxiliary tasks learned via meta-gradients outperform hand-designed tasks like reward prediction and pixel control in several domains.
• Joint learning with discovered GVFs can improve data efficiency relative to baselines, with gains varying by game and task difficulty.
• Using area under the meta-loss curve provides more stable learning than last-batch meta-loss in their experiments.
• Too many GVF questions or very long meta-unrolls can hamper training efficiency or performance, indicating the need for careful hyperparameter choices.