[Paper Review] Guarantees for Epsilon-Greedy Reinforcement Learning with Function Approximation

Christoph Dann, Yishay Mansour+3 more|arXiv (Cornell University)|Jun 19, 2022

Advanced Bandit Algorithms Research26 citations

TL;DR

The paper introduces a framework and theory for myopic exploration with epsilon-greedy policies in episodic MDPs, defining a myopic exploration gap and providing upper and lower bounds on sample complexity and regret under bounded Bellman Eluder dimension.

ABSTRACT

Myopic exploration policies such as epsilon-greedy, softmax, or Gaussian noise fail to explore efficiently in some reinforcement learning tasks and yet, they perform well in many others. In fact, in practice, they are often selected as the top choices, due to their simplicity. But, for what tasks do such policies succeed? Can we give theoretical guarantees for their favorable performance? These crucial questions have been scarcely investigated, despite the prominent practical importance of these policies. This paper presents a theoretical analysis of such policies and provides the first regret and sample-complexity bounds for reinforcement learning with myopic exploration. Our results apply to value-function-based algorithms in episodic MDPs with bounded Bellman Eluder dimension. We propose a new complexity measure called myopic exploration gap, denoted by alpha, that captures a structural property of the MDP, the exploration policy and the given value function class. We show that the sample-complexity of myopic exploration scales quadratically with the inverse of this quantity, 1 / alpha^2. We further demonstrate through concrete examples that myopic exploration gap is indeed favorable in several tasks where myopic exploration succeeds, due to the corresponding dynamics and reward structure.

Motivation & Objective

• Introduce a framework to analyze value-function based RL with myopic exploration.
• Define the myopic exploration gap to capture how easily suboptimal functions are identified by myopic policies.
• Derive sample complexity and regret bounds for epsilon-greedy RL with function approximation under the Bellman Eluder dimension.
• Show conditions under which myopic exploration is favorable and provide lower bounds to demonstrate tightness.

Proposed method

• Propose a least-squares regression-based algorithm (Algorithm 1) that learns Q-functions by backward induction with data collected under a myopic exploration policy.
• Define a new complexity measure, the myopic exploration gap alpha(f, F, Pi', expl, M), and its radius c(f, F, Pi', expl, M).
• Use a structured analysis linking the gap to the number of episodes needed to rule out suboptimal F' subset of F, via the Bellman Eluder dimension d and covering numbers.
• Provide the first regret and sample-complexity bounds for RL with myopic exploration in terms of alpha and c.
• Present both a general upper bound (Theorem 1) and a matching lower bound, showing tight dependence on alpha and the dimension.

Experimental results

Research questions

• RQ1 When do myopic exploration policies like epsilon-greedy yield sample-efficient learning with function approximation?
• RQ2 How does the myopic exploration gap quantify the ease of identifying suboptimal value functions under myopic policies?
• RQ3 What are the sample complexity and regret guarantees for epsilon-greedy RL in episodic MDPs with bounded Bellman Eluder dimension?
• RQ4 Under what structural conditions of the MDP (dynamics and rewards) is myopic exploration particularly effective?

Key findings

• A new myopic exploration gap alpha(f, F, Pi', expl, M) is defined to capture how easily a myopic exploration strategy identifies that a candidate value function is sub-optimal.
• The paper proves a sample complexity upper bound showing the number of episodes with suboptimal f is O((log c(F',F)) / alpha(F',F)^2 · H^2 · d · log factors).
• An almost matching lower bound of Omega(d / alpha(F',F)^2) is established, indicating tight dependence on the myopic exploration gap and Bellman Eluder dimension.
• Corollaries show that with favorable dynamics (e.g., small multiplicative action variation) or contextual bandit structure (delta_P = 1), the gap is large and learning is faster.
• Dense reward shaping can improve or hinder myopic exploration depending on how it alters the informative samples under the exploration policy.
• Potential-based reward shaping does not affect the gaps in the tabular setting, due to identical policy returns under shaping.