[Paper Review] Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

Junhyuk Oh, Satinder Singh+2 more|arXiv (Cornell University)|Jun 15, 2017

Reinforcement Learning in Robotics39 references114 citations

TL;DR

The paper proposes a two-stage hierarchical RL framework: learning parameterized skills with an analogy-making objective to generalize to unseen subtasks, followed by a meta-controller that executes instruction sequences and can interrupt/Subtask update with a novel large-time-scale learning mechanism.

ABSTRACT

As a step towards developing zero-shot task generalization capabilities in reinforcement learning (RL), we introduce a new RL problem where the agent should learn to execute sequences of instructions after learning useful skills that solve subtasks. In this problem, we consider two types of generalizations: to previously unseen instructions and to longer sequences of instructions. For generalization over unseen instructions, we propose a new objective which encourages learning correspondences between similar subtasks by making analogies. For generalization over sequential instructions, we present a hierarchical architecture where a meta controller learns to use the acquired skills for executing the instructions. To deal with delayed reward, we propose a new neural architecture in the meta controller that learns when to update the subtask, which makes learning more efficient. Experimental results on a stochastic 3D domain show that the proposed ideas are crucial for generalization to longer instructions as well as unseen instructions.

Motivation & Objective

• Develop zero-shot generalization to unseen subtasks within RL by learning analogies between subtasks.
• Enable generalization to longer sequences of instructions not seen during training.
• Handle delayed rewards where success is only granted after completing all instructions.
• Support interruptions from unexpected events by allowing dynamic subtask updates.
• Provide a hierarchical architecture that composes pre-learned subtasks to execute instruction lists.

Proposed method

• Introduce parameterized skills that map observations and task parameters to actions and a termination signal, enabling multi-task policy learning.
• Propose an analogy-making objective with contrastive losses to shape the task-embedding space and enable generalization to unseen subtasks (similarity, dissimilarity, and difference constraints).
• Train the parameterized skill with actor-critic methods (A2C/GAE) and optionally policy distillation to improve transfer.
• Develop a two-stage architecture: a generative skill learner for subtasks, and a meta controller that reads instruction lists and selects subtask parameters.
• Design a novel meta-controller with a context LSTM that incorporates the previous subtask and a termination signal, and a subtask updater that uses a soft-update mechanism to operate at a larger time-scale.
• Implement a memory-enabled instruction updater that retrieves and shifts instructions via a soft-attention over an instruction sequence, enabling one-step updates to subtask parameters.

Experimental results

Research questions

• RQ1 Can the proposed analogy-making objective enable zero-shot generalization to unseen subtasks within a parameterized skill?
• RQ2 Does generalization extend to unseen or longer instruction sequences when using the hierarchical meta-controller?
• RQ3 Does the meta-controller’s ability to decide when to update subtasks improve performance under delayed reward and interruptions?
• RQ4 How does operating at a large time-scale (via soft updates) affect learning efficiency and responsiveness to unexpected events?
• RQ5 Do the proposed methods outperform non-hierarchical baselines and other hierarchical RL variants on instruction execution tasks?

Key findings

• Analogy-based generalization enables successful transfer to unseen parameterized subtasks across several generalization scenarios.
• The hierarchical architecture with a meta-controller outperforms several baselines in executing instruction sequences, including unseen sequences.
• Learning when to update the subtask (dynamic time-scale) significantly improves performance under delayed rewards and interruptions, compared to updating at every step or only on termination.
• The large-time-scale meta-controller (with soft updates) achieves better training efficiency and robustness to unexpected events than fixed-timescale variants.
• In experiments in a Minecraft-like 3D domain, the approach generalizes to longer instruction sequences and unseen instructions, with notable gains when using the dynamic update mechanism.