[Paper Review] Implementing the Deep Q-Network

Melrose Roderick, James MacGlashan+1 more|arXiv (Cornell University)|Nov 20, 2017

Big Data and Business IntelligenceBusiness, Management and Accounting4 references56 citations

TL;DR

This paper reproduces the Deep Q-Network (DQN) results, highlights undocumented implementation details, and provides a faster, flexible DQN implementation with insights on stability and performance.

ABSTRACT

The Deep Q-Network proposed by Mnih et al. [2015] has become a benchmark and building point for much deep reinforcement learning research. However, replicating results for complex systems is often challenging since original scientific publications are not always able to describe in detail every important parameter setting and software engineering solution. In this paper, we present results from our work reproducing the results of the DQN paper. We highlight key areas in the implementation that were not covered in great detail in the original paper to make it easier for researchers to replicate these results, including termination conditions and gradient descent algorithms. Finally, we discuss methods for improving the computational performance and provide our own implementation that is designed to work with a range of domains, and not just the original Arcade Learning Environment [Bellemare et al., 2013].

Motivation & Objective

• Reproduce the DQN results on Atari games to verify performance.
• Identify and document critical but under-described implementation details.
• Improve computational efficiency and generality of the DQN implementation across domains.
• Provide an open-source implementation to facilitate replication and extension.

Proposed method

• Present the DQN algorithm with a CNN function approximator for Q-values.
• Use experience replay with random minibatches to decorrelate training data.
• Employ a target network with periodic updates to stabilize learning.
• Describe practical implementation details (e.g., no-op starts, frame history, pretraining steps).
• Compare gradient descent optimizer choices and their impact on training stability and speed.
• Show performance improvements and speedups over the original DQN implementation.

Experimental results

Research questions

• RQ1 What implementation choices are essential for replicating DQN performance beyond the original paper?
• RQ2 How do practical details (e.g., termination conditions, gradient descent variants) affect stability and learning outcomes?
• RQ3 Can the DQN implementation be optimized for speed while preserving results across domains beyond Atari?
• RQ4 What are the observed phenomena (e.g., catastrophic forgetting) and how can they be mitigated?

Key findings

• The authors’ implementation runs approximately 4x faster than the original DeepMind Atari DQN implementation.
• Their DQN achieves Pong: 19.7 (±1.1) vs 18.9 (±1.3); Breakout: 339.3 (±86.1) vs 401.2 (±26.9); Seaquest: 6309 (±1027) vs 5286 (±1310).
• Inclusion of training details such as end-of-life termination conditions and gradient descent optimization significantly affect performance and stability.
• Using end-of-life termination improves early training stability and overall performance, especially in complex games like Seaquest.
• Gradient descent choice (RMSProp variant) and tuning impact learning speed and stability; preallocation and cuDNN acceleration contribute to speedups.
• The paper discusses the phenomenon of fluctuating performance (catastrophic forgetting) and mitigation strategies like saving best-performing parameters.