[Paper Review] Beyond No Regret: Instance-Dependent PAC Reinforcement Learning

Andrew Wagenmaker, Max Simchowitz+1 more|arXiv (Cornell University)|Aug 5, 2021

Advanced Bandit Algorithms Research36 references4 citations

TL;DR

This paper introduces a new instance-dependent sample complexity measure, gap-visitation complexity, for PAC tabular reinforcement learning, and proposes the Moca algorithm—a planning-based method that achieves nearly minimax optimal sample complexity by enabling aggressive exploration. It demonstrates that low-regret algorithms cannot achieve this complexity, revealing a fundamental tradeoff between regret minimization and instance-optimal policy identification.

ABSTRACT

The theory of reinforcement learning has focused on two fundamental problems: achieving low regret, and identifying $ε$-optimal policies. While a simple reduction allows one to apply a low-regret algorithm to obtain an $ε$-optimal policy and achieve the worst-case optimal rate, it is unknown whether low-regret algorithms can obtain the instance-optimal rate for policy identification. We show this is not possible -- there exists a fundamental tradeoff between achieving low regret and identifying an $ε$-optimal policy at the instance-optimal rate. Motivated by our negative finding, we propose a new measure of instance-dependent sample complexity for PAC tabular reinforcement learning which explicitly accounts for the attainable state visitation distributions in the underlying MDP. We then propose and analyze a novel, planning-based algorithm which attains this sample complexity -- yielding a complexity which scales with the suboptimality gaps and the "reachability" of a state. We show our algorithm is nearly minimax optimal, and on several examples that our instance-dependent sample complexity offers significant improvements over worst-case bounds.

Motivation & Objective

• To investigate whether online-to-batch reductions from low-regret algorithms yield tight instance-dependent guarantees in non-generative, tabular PAC reinforcement learning.
• To identify the limitations of optimistic, regret-minimizing algorithms in achieving instance-optimal sample complexity for policy identification.
• To develop a new instance-dependent measure of complexity that accounts for state visitation distributions and suboptimality gaps in MDPs.
• To design and analyze a planning-based algorithm that achieves the proposed instance-dependent complexity bound.
• To demonstrate that more aggressive, non-optimistic exploration strategies are necessary for optimal policy identification in tabular MDPs.

Proposed method

• Proposes a novel instance-dependent complexity measure called gap-visitation complexity, defined as a sum over steps, states, and actions involving visitation probabilities, suboptimality gaps, and reachability.
• Introduces the Moca algorithm, a planning-based method that uses aggressive exploration to reach states of interest quickly, combined with Monte Carlo estimation and action-elimination to identify suboptimal actions.
• Employs a non-optimistic exploration strategy that prioritizes reaching and evaluating potentially suboptimal states to accelerate policy identification.
• Uses a convex combination of policies to model state visitation distributions, leveraging Caratheodory’s theorem to bound the number of required policies.
• Applies information-theoretic lower bounds via KL divergence and change-of-measure arguments to derive fundamental limits on sample complexity.
• Establishes that the algorithm’s sample complexity scales as Õ(C(M,ε)·log(1/δ)), where C(M,ε) is the gap-visitation complexity, and shows this is nearly minimax optimal.

Experimental results

Research questions

• RQ1 Can low-regret algorithms achieve instance-optimal sample complexity for PAC policy identification in tabular MDPs?
• RQ2 What is the fundamental tradeoff between achieving low regret and identifying an ϵ-optimal policy at the instance-optimal rate?
• RQ3 Is there a new measure of sample complexity in PAC RL that captures the interplay between suboptimality gaps and state reachability?
• RQ4 Can a non-optimistic, planning-based algorithm outperform optimistic, regret-minimizing approaches in terms of instance-dependent sample complexity?
• RQ5 What are the fundamental limits of sample complexity for δ-correct policy identification in tabular MDPs?

Key findings

• The gap-visitation complexity measure C(M,ε) is no larger than the minimax optimal PAC rate and matches the instance-optimal complexity in certain cases.
• The Moca algorithm achieves δ-correct policy identification with Õ(C(M,ε)·log(1/δ)) episodes, making it nearly minimax optimal.
• Low-regret algorithms cannot achieve the proposed gap-visitation complexity and can be arbitrarily worse in terms of sample complexity, especially when suboptimal actions are rarely visited.
• In the motivating example, Moca identifies the optimal action in s₂ in Õ(1/Δ²) episodes, while a low-regret algorithm requires Ω(1/(pΔ²)) episodes, showing an arbitrary improvement for small p.
• The lower bound shows that any δ-correct algorithm must satisfy E[τ] ≥ (1/(6·gap(s,a)²·w^π_h(s,a)))·log(1/(2.4δ)) for all s,a, establishing a fundamental limit on sample efficiency.
• The analysis confirms that the optimal policy is not always found by optimistic exploration, and that non-optimistic, reachability-aware planning is essential for instance-optimal learning.