[Paper Review] IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Lasse Espeholt, Hubert Soyer+10 more|arXiv (Cornell University)|Feb 5, 2018

Reinforcement Learning in Robotics3 references611 citations

TL;DR

IMPALA introduces a scalable distributed deep RL agent with decoupled actors and learners and a V-trace off-policy correction, achieving high data throughput and strong multi-task performance on DMLab-30 and Atari-57.

ABSTRACT

In this work we aim to solve a large collection of tasks using a single reinforcement learning agent with a single set of parameters. A key challenge is to handle the increased amount of data and extended training time. We have developed a new distributed agent IMPALA (Importance Weighted Actor-Learner Architecture) that not only uses resources more efficiently in single-machine training but also scales to thousands of machines without sacrificing data efficiency or resource utilisation. We achieve stable learning at high throughput by combining decoupled acting and learning with a novel off-policy correction method called V-trace. We demonstrate the effectiveness of IMPALA for multi-task reinforcement learning on DMLab-30 (a set of 30 tasks from the DeepMind Lab environment (Beattie et al., 2016)) and Atari-57 (all available Atari games in Arcade Learning Environment (Bellemare et al., 2013a)). Our results show that IMPALA is able to achieve better performance than previous agents with less data, and crucially exhibits positive transfer between tasks as a result of its multi-task approach.

Motivation & Objective

• Develop a single, scalable reinforcement learning agent capable of mastering a large collection of tasks.
• Enable efficient use of compute across many machines without sacrificing data efficiency or stability.
• Introduce a principled off-policy correction to handle lag between actors and learners.

Proposed method

• Propose the IMPALA architecture that decouples acting from learning, with trajectories sent from multiple actors to a centralized learner.
• Use V-trace, an off-policy actor-critic algorithm, to correct for policy lag between behaviour and target policies.
• Utilize synchronous parameter updates across distributed learners and GPU-accelerated mini-batch updates on the learner.
• Apply architectural and TensorFlow-based optimizations (e.g., time-folding, XLA, cuDNN) to maximize throughput.
• Evaluate with two model architectures (shallow LSTM and deep residual networks) on multi-task and single-task benchmarks.

Experimental results

Research questions

• RQ1 Can a single agent with one set of parameters learn multiple diverse tasks efficiently?
• RQ2 How does decoupling actors and learners affect data throughput and data efficiency at scale?
• RQ3 Does V-trace provide robust off-policy correction across varying lag between actors and learners?
• RQ4 What is the impact of multi-task training on transfer between tasks compared to single-task training?

Key findings

• IMPALA achieves very high data throughput, up to 250,000 frames per second, outperforming single-machine A3C by over 30x.
• IMPALA shows better data efficiency and robustness to hyperparameters compared to A3C-based agents.
• Multi-task training with IMPALA yields positive transfer and superior performance on DMLab-30 compared to task-specific experts.
• On Atari-57, IMPALA (deep, multi-task) approaches competitive performance with expert baselines, achieving a 59.7% median human-normalized score.
• Across tasks, V-trace provides stable learning with off-policy correction, especially when experience replay is used.
• Deep IMPALA with multi-task training converges faster and to higher accuracy than distributed A3C baselines.