[Paper Review] Accelerated Methods for Deep Reinforcement Learning

Adam Stooke, Pieter Abbeel|arXiv (Cornell University)|Mar 7, 2018

Reinforcement Learning in RoboticsComputer Science19 references97 citations

TL;DR

The paper introduces a unified, GPU-accelerated, multi-simulator framework to parallelize both policy-gradient and Q-value methods, enabling large batch sizes and fast learning on CPUs+GPUs, demonstrated on Atari/AE with significant wall-clock speedups.

ABSTRACT

Deep reinforcement learning (RL) has achieved many recent successes, yet experiment turn-around time remains a key bottleneck in research and in practice. We investigate how to optimize existing deep RL algorithms for modern computers, specifically for a combination of CPUs and GPUs. We confirm that both policy gradient and Q-value learning algorithms can be adapted to learn using many parallel simulator instances. We further find it possible to train using batch sizes considerably larger than are standard, without negatively affecting sample complexity or final performance. We leverage these facts to build a unified framework for parallelization that dramatically hastens experiments in both classes of algorithm. All neural network computations use GPUs, accelerating both data collection and training. Our results include using an entire DGX-1 to learn successful strategies in Atari games in mere minutes, using both synchronous and asynchronous algorithms.

Motivation & Objective

• Motivate and address the bottleneck of experiment turn-around time in deep RL by leveraging modern multi-CPU/GPU hardware.

Proposed method

• Propose a unified parallel RL framework that uses GPUs for both inference and training across multiple algorithms.
• Implement multi-GPU synchronized and asynchronous optimization for A2C, PPO, DQN and variants.
• Enable learning with much larger batch sizes than standard while preserving sample efficiency and final performance.
• Demonstrate scalable sampling with batched inferences over many parallel simulators.
• Evaluate on Atari via Arcade Learning Environment to measure throughput, speedups, and learning quality.

Experimental results

Research questions

• RQ1 Can policy-gradient and Q-value methods be adapted to learn efficiently with many parallel simulator instances without loss of performance?
• RQ2 How large can training batch sizes be while maintaining sample efficiency and final outcomes?
• RQ3 What are the speedups and bottlenecks when scaling deep RL algorithms across multiple GPUs with synchronized vs asynchronous updates?
• RQ4 How does parallel sampling impact learning stability and sample complexity in practice across different algorithms?

Key findings

• Synchronized sampling with batched inferences can achieve high hardware utilization, attaining over 35,000 samples per second on an 8-GPU server for Breakout.
• Policy-gradient and Q-learning algorithms can be adapted to learn with many parallel simulators without substantial degradation in game scores or sample efficiency across A2C, A3C, PPO, APPO, and DQN variants.
• Large training batches up to several hundreds to thousands of experiences per update can accelerate learning; some algorithms retain performance with batch sizes as large as 2,048 for Categorical DQN, while others show limits at very large batches.
• Asynchronous and synchronous multi-GPU approaches yield substantial wall-clock speedups, with policy-gradient methods completing meaningful learning in minutes rather than hours.
• For Atari experiments, several configurations achieve orders-of-magnitude reductions in training time (e.g., A2C/A3C completing 50M steps in hours rather than days) while maintaining comparable human-normalized scores.