[Paper Review] Sample complexity of episodic fixed-horizon reinforcement learning

Christoph Dann, Emma Brunskill|arXiv (Cornell University)|Dec 7, 2015

Reinforcement Learning in Robotics21 references85 citations

TL;DR

This paper establishes tight sample complexity bounds for episodic fixed-horizon reinforcement learning, deriving matching upper and lower bounds of O(|S|²|A|H²/∊² ln 1/δ) and Ω(|S||A|H²/∊² ln 1/δ + c), respectively, up to logarithmic factors and a linear |S| dependency. It improves upon prior H³ time-horizon dependencies using Bernstein's inequality.

ABSTRACT

Recently, there has been significant progress in understanding reinforcement learning in discounted infinite-horizon Markov decision processes (MDPs) by deriving tight sample complexity bounds. However, in many real-world applications, an interactive learning agent operates for a fixed or bounded period of time, for example tutoring students for exams or handling customer service requests. Such scenarios can often be better treated as episodic fixed-horizon MDPs, for which only looser bounds on the sample complexity exist. A natural notion of sample complexity in this setting is the number of episodes required to guarantee a certain performance with high probability (PAC guarantee). In this paper, we derive an upper PAC bound O(|S|2|A|H2/∊2 ln 1/δ) and a lower PAC bound Ω(|S||A|H2/∊2 ln 1/δ+c) that match up to log-terms and an additional linear dependency on the number of states |S|. The lower bound is the first of its kind for this setting. Our upper bound leverages Bernstein's inequality to improve on previous bounds for episodic finite-horizon MDPs which have a time-horizon dependency of at least H3.

Motivation & Objective

• To close the gap between existing upper and lower bounds on sample complexity in episodic fixed-horizon MDPs.
• To establish the first known lower bound for sample complexity in fixed-horizon episodic reinforcement learning.
• To improve upon prior upper bounds that scale with H³ by leveraging Bernstein's inequality.
• To provide a PAC learning guarantee for fixed-horizon MDPs with tight dependence on state space |S|, action space |A|, horizon H, and error ∊.

Proposed method

• Derives an upper PAC bound using Bernstein's inequality to tighten the dependence on the time horizon H.
• Constructs a lower bound instance to prove the first known Ω(|S||A|H²/∊² ln 1/δ + c) sample complexity lower bound for fixed-horizon MDPs.
• Analyzes the interaction between episode length H, state space size |S|, and action space |A| in the context of PAC learning.
• Uses concentration inequalities to bound estimation errors in value function approximation over fixed-horizon episodes.
• Compares the derived bounds with prior results to demonstrate improvement in H-dependency from H³ to H².

Experimental results

Research questions

• RQ1 What is the tightest possible upper bound on the number of episodes required to achieve a PAC guarantee in fixed-horizon MDPs?
• RQ2 What is the fundamental lower bound on sample complexity for fixed-horizon episodic reinforcement learning?
• RQ3 How does the sample complexity scale with the number of states |S|, actions |A|, horizon H, and desired accuracy ∊?
• RQ4 Can the H³ dependency in prior upper bounds be improved, and if so, by what technique?

Key findings

• The upper bound on sample complexity is O(|S|²|A|H²/∊² ln 1/δ), improving upon prior H³ dependencies via Bernstein's inequality.
• The first known lower bound for fixed-horizon MDPs is Ω(|S||A|H²/∊² ln 1/δ + c), matching the upper bound up to logarithmic factors.
• The upper and lower bounds match in their dependence on |S|, |A|, H, ∊, and δ, up to log terms and a linear |S| factor.
• The improvement in H-dependency from H³ to H² demonstrates a significant tightening of theoretical guarantees for fixed-horizon RL.