[Paper Review] A Deep Hierarchical Approach to Lifelong Learning in Minecraft

Chen Tessler, Shahar Givony+3 more|arXiv (Cornell University)|Apr 25, 2016

Reinforcement Learning in Robotics33 references141 citations

TL;DR

The paper introduces a Hierarchical Deep Reinforcement Learning Network (H-DRLN) that reuses pre-trained Deep Skill Networks (DSNs) via a DSN array or a distilled multi-skill network, enabling scalable lifelong learning in Minecraft and outperforming vanilla DQN in sub-domains. It also proposes a novel Skill Bellman equation and Skill Experience Replay to train with temporally extended actions.

ABSTRACT

We propose a lifelong learning system that has the ability to reuse and transfer knowledge from one task to another while efficiently retaining the previously learned knowledge-base. Knowledge is transferred by learning reusable skills to solve tasks in Minecraft, a popular video game which is an unsolved and high-dimensional lifelong learning problem. These reusable skills, which we refer to as Deep Skill Networks, are then incorporated into our novel Hierarchical Deep Reinforcement Learning Network (H-DRLN) architecture using two techniques: (1) a deep skill array and (2) skill distillation, our novel variation of policy distillation (Rusu et. al. 2015) for learning skills. Skill distillation enables the HDRLN to efficiently retain knowledge and therefore scale in lifelong learning, by accumulating knowledge and encapsulating multiple reusable skills into a single distilled network. The H-DRLN exhibits superior performance and lower learning sample complexity compared to the regular Deep Q Network (Mnih et. al. 2015) in sub-domains of Minecraft.

Motivation & Objective

• Motivate lifelong learning where agents continually acquire, retain, and transfer knowledge across tasks in high-dimensional environments like Minecraft.
• Introduce reusable skills (DSNs) and a hierarchical controller to efficiently solve new tasks by selective transfer.
• Develop scalable mechanisms (DSN array and distilled multi-skill network) to retain and combine multiple skills via skill distillation.
• Demonstrate improved learning performance and faster convergence compared to vanilla DQN in Minecraft sub-domains.
• Provide empirical evidence of knowledge transfer between related tasks without additional learning.

Proposed method

• Define skills as temporally extended actions (options) with initiation sets, intra-skill policies, and termination conditions.
• Extend DQN to Hierarchical Deep RL Network (H-DRLN) with a Deep Skill Module that either stores an array of DSNs or a distilled multi-skill network.
• Introduce Skill Bellman equation and Skill Experience Replay to handle SMDP-based learning with skills.
• Train DSNs on sub-domains using a vanilla DQN variant and integrate them into H-DRLN for new tasks.
• Apply policy distillation to fuse multiple DSNs into a single distilled network, enabling scalable knowledge retention.
• Evaluate on Minecraft sub-domains and composite domains to compare against vanilla DQN and DDQN.

Experimental results

Research questions

• RQ1 Can pre-learned DSNs learned in simple sub-domains be reused to solve more complex tasks in Minecraft?
• RQ2 Does the H-DRLN enable more data-efficient learning and higher performance than standard DQN/DDQN in hierarchical tasks?
• RQ3 Can multiple DSNs be distilled into a single scalable network without losing task-specific capabilities?
• RQ4 Does knowledge transfer occur between related tasks without additional training, and how does skill usage evolve during learning?

Key findings

• H-DRLN with DSNs achieves faster convergence and higher performance than vanilla DQN in Minecraft sub-domains.
• In the two-room domain, vanilla DQN reaches 50% success, DSN alone 67.65%, H-DRLN_START 73.08%, and H-DRLN_END 76% success.
• Distilled multi-skill network attains up to 94±4% success across rooms, outperforming DDQN baselines.
• Skill usage grows mid-training (around epoch 50) and then stabilizes as primitive actions are used to refine policies, yet skills significantly accelerate learning.
• DSNs trained on one-room domains can transfer to related tasks without further training, yielding higher rewards than a DQN trained directly on the target task.
• Training with skills reduces sample complexity and improves exploration by enabling temporally extended actions.