[Paper Review] Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms

Chi Jin, Qinghua Liu+1 more|arXiv (Cornell University)|Feb 1, 2021

Machine Learning and Algorithms48 references29 citations

TL;DR

Introduces Bellman Eluder (BE) dimension as a unifying complexity measure for RL with function approximation, showing low BE dimension subsumes many tractable RL classes, and presents the Golf optimization-based algorithm (plus Olive) with polynomial sample complexity independent of state-action size.

ABSTRACT

Finding the minimal structural assumptions that empower sample-efficient learning is one of the most important research directions in Reinforcement Learning (RL). This paper advances our understanding of this fundamental question by introducing a new complexity measure -- Bellman Eluder (BE) dimension. We show that the family of RL problems of low BE dimension is remarkably rich, which subsumes a vast majority of existing tractable RL problems including but not limited to tabular MDPs, linear MDPs, reactive POMDPs, low Bellman rank problems as well as low Eluder dimension problems. This paper further designs a new optimization-based algorithm -- GOLF, and reanalyzes a hypothesis elimination-based algorithm -- OLIVE (proposed in Jiang et al., 2017). We prove that both algorithms learn the near-optimal policies of low BE dimension problems in a number of samples that is polynomial in all relevant parameters, but independent of the size of state-action space. Our regret and sample complexity results match or improve the best existing results for several well-known subclasses of low BE dimension problems.

Motivation & Objective

• Identify minimal structural assumptions enabling sample-efficient RL with function approximation.
• Show that low BE dimension encompasses many known tractable RL classes (e.g., Bellman rank, Eluder dimension) and introduces new ones.
• Design algorithms with provable sample efficiency that are independent of state-action space size.

Proposed method

• Define Bellman Eluder (BE) dimension as the distributional Eluder dimension of Bellman residuals across steps.
• Propose the Golf algorithm, an optimization-based method that selects optimistic value functions and updates a confidence set via local regression constraints on collected data.
• Use a generalized completeness assumption and realizability to guarantee regret and sample complexity bounds.
• Reanalyze an existing hypothesis elimination algorithm Olive within the BE framework and compare performance.
• Establish connections showing BE dimension subsumes low Bellman rank and low Eluder dimension, and includes new classes like kernel reactive POMDPs.

Experimental results

Research questions

• RQ1 What is the minimal structural complexity that enables sample-efficient RL with function approximation?
• RQ2 Does BE dimension unify and extend existing tractable RL classes such as low Bellman rank and low Eluder dimension?
• RQ3 Can practical algorithms achieve poly-time sample complexity independent of state-action space under BE-based assumptions?
• RQ4 How do Golf and Olive perform under BE-based assumptions relative to existing results?

Key findings

• Low BE dimension defines a rich class of RL problems that includes many known tractable models and new ones.
• Golf achieves regret and sample complexity bounds that scale polynomially with BE dimension and horizon, and are independent of state-action space size.
• Golf attains near-optimal policies under realizability and generalized completeness with sample complexity tied to BE dimension and covering numbers.
• Olive can also learn low BE dimension problems but with worse sample complexity compared to Golf under similar assumptions.
• BE dimension subsumes low Bellman rank and low Eluder dimension, and there exist problems with low BE dimension not covered by the union of those two classes.