[Paper Review] Is Q-learning Provably Efficient?

Chi Jin, Zeyuan Allen-Zhu+2 more|arXiv (Cornell University)|Jan 1, 2018

Advanced Bandit Algorithms Research337 citations

TL;DR

This paper proves that Q-learning with UCB exploration achieves regret of $ O(\sqrt{H^3 SAT}) $ in episodic MDPs, matching the optimal regret up to a $ \sqrt{H} $ factor—establishing the first provable sample efficiency for a classical model-free RL algorithm without requiring a simulator. The result confirms Q-learning's theoretical sample efficiency in tabular settings.

ABSTRACT

Model-free reinforcement learning (RL) algorithms directly parameterize and update value functions or policies, bypassing the modeling of the environment. They are typically simpler, more flexible to use, and thus more prevalent in modern deep RL than model-based approaches. However, empirical work has suggested that they require large numbers of samples to learn. The theoretical question of whether not model-free algorithms are in fact \emph{sample efficient} is one of the most fundamental questions in RL. The problem is unsolved even in the basic scenario with finitely many states and actions. We prove that, in an episodic MDP setting, Q-learning with UCB exploration achieves regret $ lO(\sqrt{H^3 SAT})$ where $S$ and $A$ are the numbers of states and actions, $H$ is the number of steps per episode, and $T$ is the total number of steps. Our regret matches the optimal regret up to a single $\sqrt{H}$ factor. Thus we establish the sample efficiency of a classical model-free approach. Moreover, to the best of our knowledge, this is the first model-free analysis to establish $\sqrt{T}$ regret \emph{without} requiring access to a ``simulator.''

Motivation & Objective

• To resolve the fundamental open question of whether model-free RL algorithms like Q-learning are provably sample efficient.
• To analyze the regret of Q-learning with UCB exploration in episodic MDPs with finite states and actions.
• To establish a theoretical bound on sample efficiency for a classical model-free algorithm without relying on a simulator.
• To close the gap between empirical performance and theoretical understanding of Q-learning in tabular settings.

Proposed method

• Uses Q-learning with Upper Confidence Bound (UCB) exploration to balance exploration and exploitation in episodic MDPs.
• Analyzes regret over T total steps, with H steps per episode, S states, and A actions.
• Applies concentration inequalities and martingale arguments to bound estimation errors and regret.
• Establishes a high-probability regret bound of $ O(\sqrt{H^3 SAT}) $, matching the information-theoretic lower bound up to a $ \sqrt{H} $ factor.
• Relies solely on online interaction with the environment, not a simulator, to derive the bound.
• Introduces a novel analysis framework that tracks uncertainty in Q-value estimates across episodes.

Experimental results

Research questions

• RQ1 Can Q-learning with UCB exploration achieve provably low regret in tabular episodic MDPs?
• RQ2 Is model-free Q-learning sample efficient without access to a simulator?
• RQ3 How close is the regret of Q-learning to the information-theoretic lower bound?
• RQ4 Does the analysis of Q-learning in the tabular setting yield $ \sqrt{T} $-dependent regret without auxiliary assumptions?

Key findings

• Q-learning with UCB exploration achieves regret of $ O(\sqrt{H^3 SAT}) $, which matches the optimal regret bound up to a $ \sqrt{H} $ factor.
• The regret bound is derived without requiring access to a simulator, making the result applicable to real online learning settings.
• This is the first model-free analysis to establish $ \sqrt{T} $-dependent regret in tabular MDPs without simulator access.
• The result confirms that Q-learning is provably sample efficient in the tabular episodic MDP setting.
• The analysis provides a theoretical foundation for the empirical success of Q-learning in deep RL, despite its sample inefficiency concerns.