[Paper Review] Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Bounds

Andrea Zanette, Emma Brunskill|arXiv (Cornell University)|Jan 1, 2019

Advanced Bandit Algorithms ResearchDecision Sciences26 references66 citations

TL;DR

The paper introduces Euler, an episodic finite-horizon RL algorithm that achieves problem-dependent regret bounds tied to the max conditional variance of next-state values, while matching worst-case bounds in general.

ABSTRACT

Strong worst-case performance bounds for episodic reinforcement learning exist but fortunately in practice RL algorithms perform much better than such bounds would predict. Algorithms and theory that provide strong problem-dependent bounds could help illuminate the key features of what makes a RL problem hard and reduce the barrier to using RL algorithms in practice. As a step towards this we derive an algorithm for finite horizon discrete MDPs and associated analysis that both yields state-of-the art worst-case regret bounds in the dominant terms and yields substantially tighter bounds if the RL environment has small environmental norm, which is a function of the variance of the next-state value functions. An important benefit of our algorithmic is that it does not require apriori knowledge of a bound on the environmental norm. As a result of our analysis, we also help address an open learning theory question~\cite{jiang2018open} about episodic MDPs with a constant upper-bound on the sum of rewards, providing a regret bound with no $H$-dependence in the leading term that scales a polynomial function of the number of episodes.

Motivation & Objective

• Motivate the need for problem-dependent regret bounds in reinforcement learning to understand problem hardness beyond worst-case analysis.
• Propose an algorithm (Euler) that adapts exploration using variance-aware bonuses without prior environment knowledge.
• Derive high-probability regret bounds that depend on the environment's variance (Q*) and show horizon-independent behavior under certain reward-bounded settings.
• Demonstrate that the approach yields tighter bounds in domains with low environmental norm and address open learning theory questions.

Proposed method

• Introduce Euler, an episodic upper-lower exploration algorithm for finite-horizon MDPs.
• Use optimism under uncertainty with a Bernstein-type bonus based on empirical variance of next-state values.
• Incorporate a correction bonus that accounts for value-function uncertainty to ensure optimism.
• Analyze regret with a decomposition into reward estimation, transition dynamics estimation/optimism, and lower-order terms.
• Bound the dominant exploration term by a problem-dependent quantity Q* and relate it to the max return G.
• Prove a worst-case bound that matches the known O(sqrt(HSAT)) rate in the dominant term.

Experimental results

Research questions

• RQ1 Can we obtain regret bounds for episodic finite-horizon MDPs that depend on problem structure rather than being purely worst-case?
• RQ2 Does an exploration bonus based on empirical Bernstein inequalities and value-function uncertainty yield tighter, environment-dependent regret bounds without prior domain knowledge?
• RQ3 How do horizon and environmental norm influence the regret bounds in finite-horizon RL?
• RQ4 Can the proposed algorithm address open questions about horizon dependence in episodic MDPs with bounded total reward?

Key findings

• Euler achieves problem-dependent regret upper bounds of the form tilde{O}( sqrt(Q*SAT) + sqrt(S)SAH^2 (sqrt{S}+sqrt{H}) ) with high probability.
• A second bound tilde{O}( sqrt(G^2/H · SAT) + sqrt(S)SAH^2 (sqrt{S}+sqrt{H}) ) is provided, often tightening the first when G is large.
• Corollaries show horizon-independent behavior in certain reward-bounded settings, matching minimax bounds in the dominant term.
• Corollary 1.1 states a worst-case bound tilde{O}( sqrt{HSAT} + sqrt{S}SAH^2 (sqrt{S}+sqrt{H}) ).
• Corollary 1.2 shows bounds using the range of successor-state values Phi_succ, independent of the full V^{*}, and without needing Phi or environmental norm.