[Paper Review] A Theoretical Analysis of Deep Q-Learning

Jianqing Fan, Zhaoran Wang+2 more|arXiv (Cornell University)|Jan 1, 2019

Reinforcement Learning in Robotics143 references131 citations

TL;DR

The paper provides the first theoretical convergence analysis of a simplification of Deep Q-Networks (DQN), deriving algorithmic and statistical rates, and extends the framework to a Minimax-DQN for zero-sum Markov games.

ABSTRACT

Despite the great empirical success of deep reinforcement learning, its theoretical foundation is less well understood. In this work, we make the first attempt to theoretically understand the deep Q-network (DQN) algorithm (Mnih et al., 2015) from both algorithmic and statistical perspectives. In specific, we focus on a slight simplification of DQN that fully captures its key features. Under mild assumptions, we establish the algorithmic and statistical rates of convergence for the action-value functions of the iterative policy sequence obtained by DQN. In particular, the statistical error characterizes the bias and variance that arise from approximating the action-value function using deep neural network, while the algorithmic error converges to zero at a geometric rate. As a byproduct, our analysis provides justifications for the techniques of experience replay and target network, which are crucial to the empirical success of DQN. Furthermore, as a simple extension of DQN, we propose the Minimax-DQN algorithm for zero-sum Markov game with two players. Borrowing the analysis of DQN, we also quantify the difference between the policies obtained by Minimax-DQN and the Nash equilibrium of the Markov game in terms of both the algorithmic and statistical rates of convergence.

Motivation & Objective

• Motivate the need for theoretical understanding of deep Q-learning (DQN) beyond empirical success.
• Analyze a tractable simplification of DQN that retains key features like experience replay and target networks.
• Establish algorithmic (convergence) and statistical (bias-variance) rates for the action-value function under neural network approximation.
• Provide theoretical justification for techniques such as experience replay and target networks.
• Extend the framework to a Minimax-DQN algorithm for two-player zero-sum Markov games and quantify suboptimality and convergence.

Proposed method

• Model DQN as neural fitted Q-iteration (FQI) with ReLU networks and a large-batch regime.
• Introduce an independence assumption to simplify experience replay, making it resemble i.i.d. sampling.
• Represent the value function with sparse ReLU networks and bound its capacity via network sparsity.
• Establish geometric convergence of the algorithmic error to zero while characterizing a statistical error from neural approximation.
• Use Hölder smoothness and composition results to analyze the approximation error of the Bellman operator with neural nets.
• Extend the analysis to Minimax-DQN by solving a Nash-equilibrium target in a zero-sum Markov game and bounding suboptimality.

Experimental results

Research questions

• RQ1 What are the algorithmic and statistical convergence properties of DQN under a tractable, theoretically analyzable setting?
• RQ2 How do experience replay and target networks contribute to the stability and convergence of DQN from a theoretical standpoint?
• RQ3 Can the DQN framework be extended to zero-sum Markov games, and what are the convergence and suboptimality guarantees?
• RQ4 What is the impact of using sparse ReLU networks and Hölder smoothness on the convergence rates of neural FQI?
• RQ5 How does the neural-FQI analysis inform the interpretation of the Bellman operator when approximated by deep networks?

Key findings

• The neural FQI algorithm with ReLU networks converges geometrically in algorithmic error to the optimal Q-function up to a statistical error caused by neural approximation and finite samples.
• Experience replay and target networks are theoretically justified as stabilizing components that align the regression objective with Bellman optimality.
• The statistical error captures the bias and variance from approximating Q* with a neural network under finite data and network capacity.
• Under mild assumptions, the action-value function estimated by the sequence QK converges up to an intrinsic error determined by the ReLU network approximation capability and sample size.
• The Minimax-DQN extension to two-player zero-sum Markov games yields similar algorithmic and statistical convergence rates, bounding suboptimality relative to the Nash equilibrium policy.