[Paper Review] Bayesian Reinforcement Learning via Deep, Sparse Sampling

Divya Grover, Debabrota Basu+1 more|arXiv (Cornell University)|Feb 7, 2019

Reinforcement Learning in Robotics21 references4 citations

TL;DR

This paper proposes DSS (Deeper Sparser Sampling), a Bayesian reinforcement learning algorithm that uses policy sampling to build deeper and sparser planning trees over belief states, enabling efficient, optimism-free exploration. It achieves state-of-the-art performance with significantly lower computational cost by sampling long-term options instead of individual actions, and provides a PAC-style theoretical bound on suboptimality relative to the Bayes-optimal policy.

ABSTRACT

We address the problem of Bayesian reinforcement learning using efficient model-based online planning. We propose an optimism-free Bayes-adaptive algorithm to induce deeper and sparser exploration with a theoretical bound on its performance relative to the Bayes optimal policy, with a lower computational complexity. The main novelty is the use of a candidate policy generator, to generate long-term options in the planning tree (over beliefs), which allows us to create much sparser and deeper trees. Experimental results on different environments show that in comparison to the state-of-the-art, our algorithm is both computationally more efficient, and obtains significantly higher reward in discrete environments.

Motivation & Objective

• To address the computational intractability of Bayes-optimal planning in Bayesian reinforcement learning by enabling deeper and sparser tree expansions.
• To eliminate reliance on optimism-in-the-face-of-uncertainty by using policy sampling to guide exploration.
• To reduce computational complexity while maintaining strong theoretical guarantees on performance relative to the Bayes-optimal policy.
• To improve sample efficiency and long-term reward accumulation in discrete MDP environments compared to existing state-of-the-art methods.

Proposed method

• The algorithm constructs a belief tree where nodes represent information states, and uses a candidate policy generator to sample entire long-horizon policies rather than individual actions.
• At each planning step, only a limited number of high-expected-reward policies are sampled and expanded, drastically reducing the branching factor and enabling deeper tree growth.
• The method employs a PAC (Probably Approximately Correct) planning framework to bound the suboptimality gap between the DSS policy and the Bayes-optimal policy.
• It uses Thompson sampling-like policy sampling from the posterior belief over MDP models to generate diverse, high-quality options for tree expansion.
• Instead of maintaining upper and lower bounds on belief nodes (as in BFS3), DSS directly samples from the posterior to select actions, simplifying computation.
• The algorithm plans over a K-step horizon, updating policies every K steps, which reduces per-step computational overhead while maintaining long-term performance.

Experimental results

Research questions

• RQ1 Can deeper and sparser planning trees be constructed using policy-level sampling to improve exploration efficiency in Bayesian RL?
• RQ2 Does eliminating optimism-in-the-face-of-uncertainty lead to better long-term performance and lower computational cost in belief MDP planning?
• RQ3 What is the theoretical suboptimality gap between the proposed DSS algorithm and the Bayes-optimal policy, and how does it scale with planning depth K?
• RQ4 How does DSS compare to state-of-the-art BAMDP algorithms like BAMCP, BFS3, and SBOSS in terms of both reward and computational efficiency?
• RQ5 Can the use of policy sampling instead of action-level sampling lead to faster convergence to the true model and higher cumulative rewards?

Key findings

• DSS outperforms all state-of-the-art algorithms in cumulative reward across all tested environments, including Chain, DoubleLoop, Grid5, Grid10, and Maze, with statistically significant gains.
• In the Maze environment, DSS achieved a mean cumulative reward of 1532.0, surpassing BAMCP (1789.4) and BFS3 (3558.7), though BFS3 exceeded the time limit, indicating DSS's superior efficiency.
• DSS achieved higher average per-step reward than all baselines, especially in early training phases, demonstrating superior exploration capabilities.
• Computationally, DSS reduced per-episode time by a significant margin compared to BFS3 and often outperformed BAMCP, even when BAMCP’s performance dropped with increased sampling.
• The algorithm’s performance plateaued quickly with increasing hyperparameters N and M, indicating fast convergence and robustness to parameter tuning.
• The theoretical analysis confirms that the suboptimality gap between DSS and the Bayes-optimal policy is bounded and decreases with increasing planning depth K, providing strong PAC-style guarantees.