[Paper Review] Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning

Emilio Parisotto, Jimmy Ba+1 more|arXiv (Cornell University)|Nov 19, 2015

Reinforcement Learning in Robotics18 references207 citations

TL;DR

Actor-Mimic proposes a deep multitask and transfer reinforcement learning method that trains a single policy network to master multiple Atari games simultaneously by mimicking expert networks using model compression. The approach enables generalization to new tasks, significantly accelerating learning in unseen environments through shared representations learned during multitask pretraining.

ABSTRACT

The ability to act in multiple environments and transfer previous knowledge to new situations can be considered a critical aspect of any intelligent agent. Towards this goal, we define a novel method of multitask and transfer learning that enables an autonomous agent to learn how to behave in multiple tasks simultaneously, and then generalize its knowledge to new domains. This method, termed "Actor-Mimic", exploits the use of deep reinforcement learning and model compression techniques to train a single policy network that learns how to act in a set of distinct tasks by using the guidance of several expert teachers. We then show that the representations learnt by the deep policy network are capable of generalizing to new tasks with no prior expert guidance, speeding up learning in novel environments. Although our method can in general be applied to a wide range of problems, we use Atari games as a testing environment to demonstrate these methods.

Motivation & Objective

• To develop a method enabling a single deep reinforcement learning agent to learn multiple tasks simultaneously.
• To enable knowledge transfer from source tasks to new, unseen target tasks using shared representations.
• To leverage model compression techniques to train a compact multitask policy network using expert guidance.
• To demonstrate that multitask pretraining improves learning speed on novel tasks compared to random initialization.

Proposed method

• The method trains a single deep policy network (the 'mimic') to imitate multiple game-specific expert networks using imitation learning.
• Model compression techniques are applied to distill expert knowledge into a shared, compact policy network.
• A feature regression objective is used to provide richer supervision than action imitation alone, improving representation learning.
• The multitask network is fine-tuned on new target tasks, demonstrating transfer learning benefits.
• The approach uses a replay memory and target networks similar to DQN for training stability.
• The method is evaluated on the Arcade Learning Environment (ALE) using Atari 2600 games.

Experimental results

Research questions

• RQ1 Can a single deep policy network be trained to perform well across multiple distinct reinforcement learning tasks using expert guidance?
• RQ2 Does multitask pretraining with Actor-Mimic lead to faster learning on new, previously unseen tasks?
• RQ3 Can the representations learned during multitask training generalize effectively to new environments?
• RQ4 How does the inclusion of intermediate feature supervision improve performance compared to action-only imitation?
• RQ5 What is the impact of task similarity on the success of transfer learning in this framework?

Key findings

• The Actor-Mimic network achieves expert-level performance on multiple Atari games using a single shared policy network.
• Multitask pretraining with Actor-Mimic significantly accelerates learning on new target tasks compared to random initialization.
• The use of feature regression as a supervision signal leads to better generalization than action-only imitation.
• The method generalizes across tasks with similar mechanics, such as Pong and Breakout, due to shared visual and structural features.
• Transfer learning success is reduced in cases of negative transfer when source and target tasks are dissimilar.
• The approach maintains model complexity comparable to a single-task DQN while learning multiple tasks simultaneously.