[Paper Review] Diagnosing Bottlenecks in Deep Q-learning Algorithms

Justin Fu, Aviral Kumar+2 more|arXiv (Cornell University)|Feb 26, 2019

Adversarial Robustness in Machine Learning34 citations

TL;DR

The paper uses a unit-testing framework with oracle solvers to dissect function approximation, sampling, and nonstationarity in Q-learning, showing large networks aid stability, replay and early stopping mitigate overfitting, and proposing an adversarial feature matching sampling method.

ABSTRACT

Q-learning methods represent a commonly used class of algorithms in reinforcement learning: they are generally efficient and simple, and can be combined readily with function approximators for deep reinforcement learning (RL). However, the behavior of Q-learning methods with function approximation is poorly understood, both theoretically and empirically. In this work, we aim to experimentally investigate potential issues in Q-learning, by means of a "unit testing" framework where we can utilize oracles to disentangle sources of error. Specifically, we investigate questions related to function approximation, sampling error and nonstationarity, and where available, verify if trends found in oracle settings hold true with modern deep RL methods. We find that large neural network architectures have many benefits with regards to learning stability; offer several practical compensations for overfitting; and develop a novel sampling method based on explicitly compensating for function approximation error that yields fair improvement on high-dimensional continuous control domains.

Motivation & Objective

• Investigate how function approximation affects convergence and suboptimality in Q-learning.
• Quantify the impact of sampling error and overfitting on Q-learning performance.
• Examine nonstationarity from moving targets and distribution shifts and their relation to learning stability.
• Explore sampling distributions and weighting schemes to improve learning efficiency and stability.

Proposed method

• Introduce Exact-FQI, Sampling-FQI, and Replay-FQI as progressively realistic Q-learning variants.
• Use a unit-testing framework with oracle dynamics and rewards to isolate error sources.
• Evaluate on tabular domains with oracle Q-values and on high-dimensional continuous control tasks.
• Measure convergence, projection bias, and distribution shift under controlled conditions.
• Test several weighting distributions (e.g., Unif, on-policy, Replay) and propose adversarial feature matching (AFM).
• Compare performance with and without replay buffers and with oracle-like early stopping.

Experimental results

Research questions

• RQ1 How does function approximation power affect convergence and bias in Q-learning?
• RQ2 What is the empirical impact of sampling error and overfitting in Q-learning frameworks?
• RQ3 Do moving targets and distribution shifts causally drive instability in practice?
• RQ4 Which sampling/weighting distributions maximize learning speed and final performance?
• RQ5 Can novel sampling schemes like adversarial feature matching improve high-dimensional Q-learning?

Key findings

• Function approximation error is not a major problem for high-capacity function approximators, and divergence is rare (0.9% in their experiments).
• Overfitting due to limited samples degrades performance, and replay buffers help mitigate it by improving coverage.
• Large neural networks yield better learning stability and final performance despite overfitting risks.
• Among sampling schemes, high-entropy and broader distributions improve performance; on-policy is not always best; replay buffers reduce distribution shift.
• Adversarial feature matching (AFM) provides a practical, high-entropy sampling approach that compensates for function approximation error and matches reported improvements in the study.