[Paper Review] Batch Value-function Approximation with Only Realizability

Tengyang Xie, Nan Jiang|arXiv (Cornell University)|Aug 11, 2020

Reinforcement Learning in Robotics39 references30 citations

TL;DR

The paper introduces BVFT, a batch reinforcement learning algorithm that achieves polynomial sample complexity for learning Q* from exploratory, polynomial-sized data under realizability, using a pairwise tournament over candidate value functions and a piecewise-constant abstraction strategy.

ABSTRACT

We make progress in a long-standing problem of batch reinforcement learning (RL): learning $Q^\star$ from an exploratory and polynomial-sized dataset, using a realizable and otherwise arbitrary function class. In fact, all existing algorithms demand function-approximation assumptions stronger than realizability, and the mounting negative evidence has led to a conjecture that sample-efficient learning is impossible in this setting (Chen and Jiang, 2019). Our algorithm, BVFT, breaks the hardness conjecture (albeit under a stronger notion of exploratory data) via a tournament procedure that reduces the learning problem to pairwise comparison, and solves the latter with the help of a state-action partition constructed from the compared functions. We also discuss how BVFT can be applied to model selection among other extensions and open problems.

Motivation & Objective

• Determine the minimal function-approximation assumption that enables polynomial sample complexity in batch RL under realizability.
• Develop an algorithm that learns Q* from exploratory batch data without stronger function-approximation assumptions.
• Provide a polynomial-sample-analysis framework that handles an unstructured function class via a piecewise-constant abstraction and tournament mechanism.
• Analyze limitations and discuss extensions such as model selection and relaxing assumptions.

Proposed method

• Introduce Batch Value-Function Tournament (BVFT) that discretizes outputs of candidate functions to form a piecewise-constant superset.
• Construct a partition of the state-action space where functions are constant, enabling a projected Bellman operator with a gamma-contraction in the L-infinity sense.
• Perform pairwise comparisons across all candidate functions using a data-dependent statistic based on the discretized projected Bellman update.
• Output the function surviving all pairwise comparisons as the learned q-function, yielding a greedy policy.
• Provide a polynomial-sample bound under Assumption 1 (exploratory data via a concentrability coefficient).
• Discuss how the approach can be extended to model selection and other extensions despite computational inefficiency for large |F|.

Experimental results

Research questions

• RQ1 Can Q* be learned in batch RL with only realizability, under exploratory data and polynomial-sized datasets?
• RQ2 How can a realizable yet arbitrary function class be leveraged to identify Q* without stronger approximation assumptions?
• RQ3 What is the sample complexity of learning a near-optimal policy when using a piecewise-constant augmentation of the function class?
• RQ4 What are the limitations of BVFT and how might they be mitigated or extended to model selection and other problems?

Key findings

• BVFT achieves a polynomial-sample learning guarantee under Assumption 1, yielding a near-optimal policy with high probability.
• The algorithm reduces learning to pairwise comparisons by discretizing function outputs and using a projected Bellman operator on a piecewise-constant class.
• A piecewise-constant abstraction allows the projected Bellman operator to be a gamma-contraction with a unique fixed point corresponding to Q*, under zero discretization error.
• The method handles approximation error in the function class via an epsilon_F term and provides a bound that includes this realizability gap.
• BVFT enumerates over the entire function class and remains feasible when the class size is polynomial, and it can be used for model selection.