[Paper Review] Posterior Sampling for Large Scale Reinforcement Learning

Georgios Theocharous, Zheng Wen+2 more|arXiv (Cornell University)|Nov 21, 2017

Advanced Bandit Algorithms ResearchDecision Sciences13 references19 citations

TL;DR

This paper proposes Deterministic Schedule Posterior Sampling Reinforcement Learning (DS-PSRL), a practical, model-independent algorithm for large-scale and continuous MDPs that uses a logarithmic, deterministic episode-switching schedule. It achieves Bayesian regret bounds independent of state space size under mild assumptions, enabling efficient generalization and outperforming state-of-the-art PSRL methods on both discrete and continuous control problems.

ABSTRACT

We propose a practical non-episodic PSRL algorithm that unlike recent state-of-the-art PSRL algorithms uses a deterministic, model-independent episode switching schedule. Our algorithm termed deterministic schedule PSRL (DS-PSRL) is efficient in terms of time, sample, and space complexity. We prove a Bayesian regret bound under mild assumptions. Our result is more generally applicable to multiple parameters and continuous state action problems. We compare our algorithm with state-of-the-art PSRL algorithms on standard discrete and continuous problems from the literature. Finally, we show how the assumptions of our algorithm satisfy a sensible parametrization for a large class of problems in sequential recommendations.

Motivation & Objective

• Address the lack of practical, scalable PSRL algorithms for non-episodic, non-resetting MDPs common in real-world applications like sequential recommendations.
• Overcome limitations of existing PSRL methods that rely on tabular representations and dynamic visitation-based episode scheduling, which are infeasible for continuous or high-dimensional state-action spaces.
• Develop a general-purpose PSRL algorithm with provable Bayesian regret guarantees applicable to parametrized MDPs, especially those with scalar parameters.
• Demonstrate the algorithm's practicality and efficiency in sample, time, and space complexity while maintaining strong theoretical performance guarantees.
• Show that the algorithm's assumptions are satisfied by a broad class of sequential recommendation systems, enabling data-efficient, non-myopic personalization.

Proposed method

• Introduce a deterministic, model-independent episode switching schedule based on logarithmic episode length increases, eliminating dependence on dynamic statistics like visitation counts.
• Sample a model from the posterior distribution at the start of each episode and execute the optimal policy for that sampled model until the episode ends.
• Use a parametric model where system dynamics depend on a scalar parameter (e.g., user propensity to listen), enabling generalization across states and actions.
• Apply Lipschitz continuity and posterior concentration assumptions to derive regret bounds independent of state space size.
• Leverage the structure of parametrized MDPs to avoid per-state-action counting, enabling scalability to continuous and high-dimensional problems.
• Use a non-episodic, continuing learning framework without state resetting, suitable for real-world applications like recommendation systems.

Experimental results

Research questions

• RQ1 Can a PSRL algorithm be designed that is both theoretically sound and practically efficient for large-scale, continuous, and non-episodic MDPs?
• RQ2 Does a deterministic episode-switching schedule—based on logarithmic episode length increases—enable strong regret guarantees without relying on dynamic statistics like visitation counts?
• RQ3 Can the regret bound of DS-PSRL be independent of the number of states when the MDP dynamics are parametrized by a scalar?
• RQ4 To what extent do the assumptions of Lipschitz dynamics and posterior concentration hold in real-world sequential recommendation systems?
• RQ5 How does DS-PSRL compare empirically to state-of-the-art PSRL algorithms in terms of sample efficiency and cumulative regret on continuous and discrete control tasks?

Key findings

• DS-PSRL achieves a Bayesian regret bound of Õ(HS√(AT)) under mild assumptions, with the bound independent of the number of states when dynamics are parametrized by a scalar.
• The algorithm outperforms state-of-the-art PSRL methods on standard discrete and continuous control problems, including the linear quadratic regulator (LQR) and POI recommendation tasks.
• Empirical results show that DS-PSRL learns optimal parameters (e.g., A*, B*) rapidly, with episode switching every time-step performing well—indicating minimal need for multi-step exploration.
• The POI recommendation model with perturbed transition probabilities satisfies both Lipschitz dynamics and posterior concentration assumptions, with ‖P(⋅|X,a,θ)−P(⋅|X,a,θ′)‖₁ ≤ (2/e)|θ−θ′|.
• The expected squared estimation error of the true parameter θ* is bounded: maxⱼ𝔼[Nⱼ₋₁|θ*−θ̃ⱼ|²] = O(1), confirming posterior concentration under mild conditions.
• The algorithm’s deterministic schedule ensures logarithmic episode length growth, enabling efficient time, sample, and space complexity, making it suitable for large-scale and continuous problems.