[Paper Review] Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learning

Christoph Dann, Tor Lattimore+1 more|arXiv (Cornell University)|Mar 22, 2017

Advanced Bandit Algorithms ResearchDecision Sciences23 references60 citations

TL;DR

The paper introduces Uniform-PAC, a framework unifying PAC and regret for episodic RL and presents UBEV, an optimistic algorithm that achieves near-optimal Uniform-PAC and regret bounds using time-uniform Law of Iterated Logarithm confidence bounds.

ABSTRACT

Statistical performance bounds for reinforcement learning (RL) algorithms can be critical for high-stakes applications like healthcare. This paper introduces a new framework for theoretically measuring the performance of such algorithms called Uniform-PAC, which is a strengthening of the classical Probably Approximately Correct (PAC) framework. In contrast to the PAC framework, the uniform version may be used to derive high probability regret guarantees and so forms a bridge between the two setups that has been missing in the literature. We demonstrate the benefits of the new framework for finite-state episodic MDPs with a new algorithm that is Uniform-PAC and simultaneously achieves optimal regret and PAC guarantees except for a factor of the horizon.

Motivation & Objective

• Motivate the need for performance guarantees that jointly bound errors across all epsilon levels in episodic RL.
• Define Uniform-PAC as a strong, time-uniform extension of PAC that implies high-probability regret bounds.
• Develop an algorithm that achieves Uniform-PAC while also delivering near-optimal PAC and regret guarantees.
• Provide theoretical analysis showing Uniform-PAC implies convergence to optimal policies with high probability.

Proposed method

• Introduce Uniform-PAC as a framework and relate it to PAC and regret guarantees.
• Propose UBEV, an optimistic RL algorithm that uses time-uniform, law-of-iterated-logarithm (LIL) confidence bounds.
• Model episodic fixed-horizon MDPs with time-dependent dynamics and use backward induction with confidence intervals on transitions and rewards.
• Employ LIL-based confidence widths phi(s,a,t) = sqrt((2 ln ln max{e,n(s,a,t)}) + ln(18SAH/δ)) / sqrt(n(s,a,t)).
• Demonstrate that UBEV achieves Uniform-PAC bounds and near-optimal regret, with sample complexity and regret dependences described in Theorem 4.

Experimental results

Research questions

• RQ1 Can Uniform-PAC provide simultaneous high-probability guarantees across all epsilon levels in episodic RL?
• RQ2 Can an algorithm be Uniform-PAC and also achieve near-optimal PAC and regret guarantees?
• RQ3 What confidence-interval construction enables uniform, time-harmless guarantees in RL?
• RQ4 How do Uniform-PAC guarantees relate to existing PAC and regret notions in episodic MDPs?
• RQ5 What are the theoretical limits of converting PAC or regret guarantees into Uniform-PAC guarantees?

Key findings

• UBEV is Uniform-PAC with a bound on the number of epsilon-errors that scales as O(SAGH^4/ε^2) times polylog factors.
• With probability at least 1−δ, UBEV ensures regret R(T) = O(H^2(√(SAT) + S^3A^2) polylog(S,A,H,T)).
• Uniform-PAC guarantees imply convergence to optimal policies with high probability and yield a Uniform High-Probability Regret bound.
• Uniform-PAC is shown to be strictly stronger than both PAC and high-probability regret, and implies both when applicable.
• UBEV uses time-uniform LIL confidence bounds that shrink as sqrt((log log n)/n), enabling uniform guarantees over all episodes.
• The bounds improve on prior MBIE-type results by reducing horizon dependence and achieving near-optimal dependence on S, A, and H.