[Paper Review] Neural Temporal-Difference and Q-Learning Provably Converge to Global Optima

Qi Cai, Zhuoran Yang+2 more|arXiv (Cornell University)|May 24, 2019

Reinforcement Learning in Robotics75 references15 citations

TL;DR

This paper proves the first nonasymptotic global convergence of neural temporal-difference (TD) and Q-learning algorithms to the global optimum of the mean-squared projected Bellman error (MSPBE). By leveraging overparametrized deep neural networks, the authors establish sublinear convergence rates—$1/T$ with population semigradients and $1/ig/ ext{sqrt}{T}$ with stochastic semigradients—enabling provable convergence despite nonconvexity and potential divergence in standard TD learning.

ABSTRACT

Temporal-difference learning (TD), coupled with neural networks, is among the most fundamental building blocks of deep reinforcement learning. However, due to the nonlinearity in value function approximation, such a coupling leads to nonconvexity and even divergence in optimization. As a result, the global convergence of neural TD remains unclear. In this paper, we prove for the first time that neural TD converges at a sublinear rate to the global optimum of the mean-squared projected Bellman error for policy evaluation. In particular, we show how such global convergence is enabled by the overparametrization of neural networks, which also plays a vital role in the empirical success of neural TD. Beyond policy evaluation, we establish the global convergence of neural (soft) Q-learning, which is further connected to that of policy gradient algorithms.

Motivation & Objective

• To close the theory-practice gap in deep reinforcement learning by rigorously justifying the empirical success of neural TD and Q-learning.
• To resolve the longstanding challenge of nonconvexity and potential divergence in neural TD due to nonlinear function approximation.
• To establish nonasymptotic global convergence of neural TD and (soft) Q-learning to the global optimum of the MSPBE objective.
• To connect neural Q-learning to policy gradient methods by proving global convergence of a variant of the policy gradient algorithm.
• To demonstrate that overparametrization enables implicit local linearization, which stabilizes training and ensures convergence.

Proposed method

• Utilizes overparametrized multi-layer neural networks to implicitly linearize the value function along the optimization path, mirroring explicit linearization in nonlinear TD.
• Analyzes neural TD using population and stochastic semigradients, proving convergence to the global optimum of the MSPBE objective.
• Introduces a notion of one-point monotonicity for semigradients in overparametrized networks, enabling convergence analysis under nonconvexity.
• Employs random feature-based approximation and concentration inequalities to bound initialization-dependent terms and control deviation from linearization.
• Establishes that the projection in MSPBE allows for rich function representation equivalent to a reproducing kernel Hilbert space under random initialization.
• Leverages the connection between (soft) Q-learning and policy gradient algorithms to extend global convergence to policy optimization.

Experimental results

Research questions

• RQ1 Can neural TD provably converge to the global optimum of the MSPBE objective despite nonconvexity and biased gradients?
• RQ2 What role does overparametrization play in stabilizing neural TD and enabling global convergence?
• RQ3 Does neural (soft) Q-learning converge globally to the optimal MSPBE solution under appropriate conditions?
• RQ4 Can the global convergence of neural Q-learning be linked to policy gradient algorithms?
• RQ5 How does the implicit local linearization from overparametrization mitigate divergence in nonlinear TD learning?

Key findings

• Neural TD converges globally to the global optimum of the MSPBE at a rate of $1/T$ when using population semigradients.
• With stochastic semigradients, neural TD converges at the rate of $1/ig/ ext{sqrt}{T}$, establishing nonasymptotic convergence under realistic online learning settings.
• Neural (soft) Q-learning converges globally to the MSPBE optimum at the same rate as neural TD, under stronger regularity conditions.
• The global convergence of neural Q-learning implies global convergence of a variant of the policy gradient algorithm, linking value-based and policy-based deep RL methods.
• Overparametrization enables implicit local linearization that avoids divergence and ensures convergence, even in the absence of explicit linearization as in nonlinear TD.
• The representation power of the neural network under random initialization matches that of a reproducing kernel Hilbert space, enabling zero MSPBE for a broad class of RL problems.