[논문 리뷰] Asynchronous Methods for Deep Reinforcement Learning

Volodymyr Mnih, Adrià Puigdomènech Badia외 6명|arXiv (Cornell University)|2016. 02. 04.

Reinforcement Learning in Robotics참고 문헌 32인용 수 1,690

한 줄 요약

The paper introduces asynchronous, parallel actor-learners to train deep reinforcement learning agents on a single CPU machine, proposing asynchronous variants of four standard RL algorithms (including A3C) that achieve strong performance across Atari, continuous control, and 3D visual navigation tasks.

ABSTRACT

We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.

연구 동기 및 목표

• Motivate a lightweight, stable deep RL framework that avoids experience replay while leveraging parallelism.
• Propose asynchronous variants of one-step Q-learning, one-step Sarsa, n-step Q-learning, and advantage actor-critic (A3C).
• Demonstrate stability, scalability, and data efficiency of asynchronous methods across discrete and continuous tasks.

제안 방법

• Use multiple CPU threads as asynchronous actor-learners updating a shared neural network model on-policy or off-policy.
• Avoid experience replay by relying on diverse exploration across parallel actors to stabilize learning.
• Apply forward-view n-step returns for updating neural networks in the asynchronous setting.
• Share optimizer statistics across threads with a Hogwild!-style update scheme.
• In A3C, jointly learn a policy and a value function with an entropy bonus to encourage exploration.
• Experiment with SGD with momentum and RMSProp, highlighting shared RMSProp statistics for robustness.

실험 결과

연구 질문

• RQ1 Can asynchronous parallel actor-learners stabilize training of deep neural network controllers without experience replay?
• RQ2 Do asynchronous variants of Q-learning, Sarsa, n-step Q-learning, and A3C work across Atari, TORCS, MuJoCo, and Labyrinth?
• RQ3 Does parallelism yield speedups and data efficiency while maintaining performance on both discrete and continuous tasks?

주요 결과

• All four asynchronous methods successfully train neural network controllers on Atari 2600 games.
• A3C achieves state-of-the-art performance on Atari, beating prior methods in half the training time using 16 CPU cores and no GPU.
• Asynchronous methods scale well with the number of parallel workers, achieving substantial speedups.
• A3C also sustains good performance on continuous control tasks using MuJoCo and on visual 3D maze navigation in Labyrinth.
• Parallel actor-learners impart a stabilizing effect on learning for value-based methods without replay.
• Training on CPU cores with A3C outperforms GPU-based DQN in several settings and provides robust learning across varied learning rates.