[Paper Review] Towards Characterizing Divergence in Deep Q-Learning

Joshua Achiam, Ethan Knight+1 more|arXiv (Cornell University)|Mar 21, 2019

Domain Adaptation and Few-Shot LearningComputer Science24 references61 citations

TL;DR

This paper analyzes divergence in Deep Q-Learning by a Taylor expansion of the update, identifies the roles of function approximation, data distribution, and bootstrapping, and proposes PreQN to stabilize learning without common tricks. It also links PreQN to natural gradient methods and demonstrates competitive performance on MuJoCo benchmarks.

ABSTRACT

Deep Q-Learning (DQL), a family of temporal difference algorithms for control, employs three techniques collectively known as the `deadly triad' in reinforcement learning: bootstrapping, off-policy learning, and function approximation. Prior work has demonstrated that together these can lead to divergence in Q-learning algorithms, but the conditions under which divergence occurs are not well-understood. In this note, we give a simple analysis based on a linear approximation to the Q-value updates, which we believe provides insight into divergence under the deadly triad. The central point in our analysis is to consider when the leading order approximation to the deep-Q update is or is not a contraction in the sup norm. Based on this analysis, we develop an algorithm which permits stable deep Q-learning for continuous control without any of the tricks conventionally used (such as target networks, adaptive gradient optimizers, or using multiple Q functions). We demonstrate that our algorithm performs above or near state-of-the-art on standard MuJoCo benchmarks from the OpenAI Gym.

Motivation & Objective

• Motivate and characterize divergence in deep Q-learning under the deadly triad (function approximation, off-policy data, bootstrapping).
• Derive a leading-order update operator to understand stability via contraction properties in the sup norm.
• Identify the roles of neural tangent kernel and data distribution in stability and generalization.
• Propose and evaluate Preconditioned Q-Networks (PreQN) to stabilize DQL without target networks or multiple Q-functions.
• Connect PreQN to natural gradient methods to explain stability under certain conditions.

Proposed method

• Perform a Taylor expansion of the DQL update to obtain a leading-order operator involving the neural tangent kernel K_theta and replay-data distribution D_rho.
• Derive an approximate update U that separates contributions from function approximation, data distribution, and bootstrapping to study contraction properties.
• Establish intuition through a sequence of update operators U1, U2, U3 and contraction criteria under sup norm.
• Introduce PreQN, a minibatch-based preconditioning of TD-errors using the inverse of a minibatch K_theta, with a line search ensuring alignment of updates with TD-errors (cosine term).
• Show equivalence between PreQN and natural gradient Q-learning under shared sample estimates (matrix forms with Phi_theta).
• Provide pseudocode for PreQN (in the style of DDPG) and discuss removal of target networks.

Experimental results

Research questions

• RQ1 Under what conditions is the leading-order DQL update a contraction in the sup norm, and how do the components of the deadly triad affect stability?
• RQ2 How does the neural tangent kernel (NTK) influence stability and generalization in DQL, and can architecture choices mitigate divergence?
• RQ3 Can a preconditioned update (PreQN) stabilize DQL without target networks or multiple Q-functions and how does it relate to natural gradient methods?
• RQ4 Do modifications to data distribution or TD-errors, as in existing methods, adequately address divergence versus addressing the NTK-driven generalization issues?
• RQ5 How does PreQN perform on continuous control benchmarks compared to TD3 and SAC, and how do activations (e.g., sin vs relu) affect stability and performance?

Key findings

• A leading-order DQL update can be analyzed via a contraction perspective in the sup norm, highlighting the roles of K_theta, D_rho, and T^*Q_theta.
• The neural tangent kernel (NTK) and the extent of generalization (off-diagonal K elements) influence stability; more aggressive generalization can harm stability.
• PreQN preconditions TD-errors by solving a minibatch least-squares problem for K_theta and updates Q with a line search to ensure alignment with TD-errors, achieving non-expansive-like behavior.
• PreQN is equivalent to natural gradient Q-learning under identical sample-based estimates, explaining observed stability without target networks in some settings.
• Empirical NTK analysis suggests sin activations can yield favorable diagonal-dominant NTK structures, with PreQN-sin outperforming or matching baselines in several MuJoCo tasks.
• PreQN demonstrates stable and competitive performance on standard MuJoCo benchmarks without target networks, though occasional divergence with relu networks indicates architecture and hyperparameter considerations remain important.