[Paper Review] Contextual Decision Processes with Low Bellman Rank are PAC-Learnable

Nan Jiang, Akshay Krishnamurthy+3 more|arXiv (Cornell University)|Oct 29, 2016

Neural Networks and Applications153 citations

TL;DR

The paper introduces Contextual Decision Processes (CDPs) and a low Bellman rank condition, then provides the Olive algorithm with PAC guarantees that learn near-optimal policies independent of context space size.

ABSTRACT

This paper studies systematic exploration for reinforcement learning with rich observations and function approximation. We introduce a new model called contextual decision processes, that unifies and generalizes most prior settings. Our first contribution is a complexity measure, the Bellman rank, that we show enables tractable learning of near-optimal behavior in these processes and is naturally small for many well-studied reinforcement learning settings. Our second contribution is a new reinforcement learning algorithm that engages in systematic exploration to learn contextual decision processes with low Bellman rank. Our algorithm provably learns near-optimal behavior with a number of samples that is polynomial in all relevant parameters but independent of the number of unique observations. The approach uses Bellman error minimization with optimistic exploration and provides new insights into efficient exploration for reinforcement learning with function approximation.

Motivation & Objective

• Motivate reinforcement learning with rich observations and function approximation under a unified CDP framework.
• Define Bellman rank as a complexity measure that captures explorable structure in CDPs.
• Propose the Olive algorithm that combines optimistic exploration with Bellman-error-based elimination.
• Prove PAC guarantees: near-optimal policy with sample complexity polynomial in M, H, K, and log factors, independent of context space size.

Proposed method

• Formalize CDPs as a general RL model that encompasses MDPs and POMDPs via contexts.
• Introduce Bellman factorization and Bellman rank to quantify exploitable structure.
• Define average Bellman error and Bellman equations for the CDP setting.
• Develop Olive (Optimism Led Iterative Value-function Elimination) that iteratively eliminates non-valid value functions based on Bellman errors.
• Provide a PAC guarantee showing sample complexity is poly(M, H, K, 1/epsilon, log N, 1/delta) and independent of the context space size.

Experimental results

Research questions

• RQ1 Can near-optimal policies be efficiently learned in CDPs with rich observations using function approximation?
• RQ2 How does the Bellman rank quantify the tractability of exploration across diverse RL settings?
• RQ3 Can a single algorithm provide PAC guarantees across MDPs, POMDPs, and related models with low Bellman rank?
• RQ4 What is the role of Bellman error minimization combined with optimistic exploration in achieving sample efficiency?

Key findings

• CDPs with low Bellman rank admit tractable, sample-efficient learning.
• Olive achieves a PAC guarantee: it finds an ε-suboptimal policy with a number of trajectories tilde-ODE poly(M, H, K, log(N/δ), 1/ε).
• The sample complexity is independent of the size of the context space, enabling effective use of rich observations and function approximation.
• The Bellman rank framework applies to tabular MDPs, low-rank MDPs, reactive POMDPs, PSRs, and even LQRs (with noted caveats for continuous actions).
• The method connects Bellman-error minimization with optimistic exploration to provide new insights into exploration under function approximation.