[Paper Review] Playing Atari with Deep Reinforcement Learning

Volodymyr Mnih, Koray Kavukcuoglu+5 more|arXiv (Cornell University)|Dec 19, 2013

Reinforcement Learning in Robotics29 references5,113 citations

TL;DR

The paper presents a deep Q-network (DQN) that learns to play Atari 2600 games from raw pixel input using experience replay and a variant of Q-learning, achieving state-of-the-art results on most games.

ABSTRACT

We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards. We apply our method to seven Atari 2600 games from the Arcade Learning Environment, with no adjustment of the architecture or learning algorithm. We find that it outperforms all previous approaches on six of the games and surpasses a human expert on three of them.

Motivation & Objective

• Demonstrate that a deep convolutional neural network can learn control policies directly from high-dimensional visual input.
• Develop and stabilize a deep Q-learning algorithm with experience replay for RL from raw pixels.
• Evaluate the approach across multiple Atari 2600 games with a single architecture and no game-specific engineering.
• Show that the method outperforms prior RL methods and approaches or surpasses human performance on several games.

Proposed method

• Use a convolutional neural network (Q-network) to approximate the action-value function Q(s,a;θ) from raw screen frames.
• Apply a variant of Q-learning with a target y = r + γ max_a' Q(s',a';θ−) and stochastic gradient descent.
• Incorporate experience replay by storing transitions (s,a,r,s') in a large replay memory and sampling minibatches for updates.
• Preprocess input frames by grayscaling, downsampling to 84x84, and stacking the last 4 frames as input to the network.
• Adopt an architecture with 4D input 84x84x4, two convolutional layers (16x8x8, stride 4; 32x4x4, stride 2) and a 256-unit fully connected layer, followed by an output unit per action.
• Train with RMSProp, use epsilon-greedy exploration, and apply frame-skipping to increase training efficiency.

Experimental results

Research questions

• RQ1 Can a single deep neural network architecture learn to play a range of Atari 2600 games using only raw pixel input and reward signals?
• RQ2 Does incorporating experience replay enable stable and data-efficient learning for deep Q-learning in high-dimensional visual domains?
• RQ3 How does a deep Q-network perform relative to prior RL methods and human players across diverse games without game-specific feature engineering?

Key findings

• The DQN outperformed all previously reported RL methods on six of the seven Atari games tested.
• The DQN surpassed a human expert on three of the seven games.
• Using a single architecture and hyperparameters across games without hand-crafted features yielded strong results.
• Experience replay and off-policy Q-learning contributed to data efficiency and training stability for large neural networks in RL.
• Training progress was monitorable via the Q-value predictions, which rose more smoothly than episodic rewards.
• The method could learn directly from raw pixel inputs to produce competitive game-playing policies.