[Paper Review] Why is Posterior Sampling Better than Optimism for Reinforcement Learning?

Ian Osband, Benjamin Van Roy|arXiv (Cornell University)|Jul 1, 2016

Advanced Bandit Algorithms ResearchDecision Sciences29 references115 citations

TL;DR

The paper shows that posterior sampling (PSRL) matches or exceeds optimistic RL in Bayesian regret, achieves an improved bound of ~O(H√SAT) for finite-horizon MDPs, and empirically outperforms OFU-based methods like UCRL2.

ABSTRACT

Computational results demonstrate that posterior sampling for reinforcement learning (PSRL) dramatically outperforms algorithms driven by optimism, such as UCRL2. We provide insight into the extent of this performance boost and the phenomenon that drives it. We leverage this insight to establish an $\ ilde{O}(H\\sqrt{SAT})$ Bayesian expected regret bound for PSRL in finite-horizon episodic Markov decision processes, where $H$ is the horizon, $S$ is the number of states, $A$ is the number of actions and $T$ is the time elapsed. This improves upon the best previous bound of $\ ilde{O}(H S \\sqrt{AT})$ for any reinforcement learning algorithm.

Motivation & Objective

• Motivate exploration-exploitation trade-offs in finite-horizon MDPs under a Bayesian framework.
• Compare posterior sampling (PSRL) with optimism-driven OFU algorithms (e.g., UCRL2) in terms of regret and sample efficiency.
• Derive Bayesian regret bounds for PSRL, improving upon previous bounds for any RL algorithm.
• Provide empirical evidence that PSRL outperforms OFU methods and discuss computational vs. statistical efficiency trade-offs.

Proposed method

• Formulates reinforcement learning as Bayesian RL with an unknown MDP treated as a random variable.
• Defines BayesRegret and relates PSRL to OFU via stochastic optimism.
• Proves BayesRegret bounds for PSRL of ~O(H√SAT) using Gaussian-Dirichlet concentration tools and a posterior sampling argument.
• Presents a conjectured tighter bound ~O(√(HSAT)) and contrasts with lower bounds; supports theory with computational experiments.
• Includes a Gaussian-PSRL variant to illustrate the conjectured improvements and computational trade-offs.

Experimental results

Research questions

• RQ1 How does PSRL perform in Bayesian regret relative to OFU-based RL algorithms?
• RQ2 Can PSRL achieve improved Bayesian regret bounds in finite-horizon, tabular MDPs compared to existing OFU approaches?
• RQ3 What are the roles of confidence-set construction and sampling in driving PSRL's performance?
• RQ4 Do empirical results support the theoretical advantages of PSRL over UCRL2 and related OFU methods?

Key findings

• PSRL matches the statistical efficiency of optimistic RL up to constant factors in Bayesian regret.
• PSRL achieves a Bayesian regret bound of Õ(H√SAT), improving on the previous Õ(HS√AT) bound for any RL algorithm.
• Empirical results show PSRL substantially outperforms OFU-based methods like UCRL2 in finite-horizon MDPs.
• The paper argues that achieving similar statistical efficiency with OFU alone may be computationally intractable, while PSRL remains tractable.
• A potential improvement conjecture suggests a tighter bound Õ(√(HSAT)) may be possible under certain analyses.