[Paper Review] Learning in POMDPs with Monte Carlo Tree Search

Sammie Katt, Frans A. Oliehoek+1 more|arXiv (Cornell University)|Jun 14, 2018

Reinforcement Learning in Robotics16 references3 citations

TL;DR

This paper introduces BA-POMCP, a Monte Carlo Tree Search extension for Bayes-Adaptive POMDPs that enables efficient online learning in partially observable environments with unknown dynamics. By leveraging root sampling, expected models, linking states, and structured model representation, BA-POMCP achieves convergence to the optimal solution and scales to problems previously intractable for BA-POMDPs.

ABSTRACT

The POMDP is a powerful framework for reasoning under outcome and information uncertainty, but constructing an accurate POMDP model is difficult. Bayes-Adaptive Partially Observable Markov Decision Processes (BA-POMDPs) extend POMDPs to allow the model to be learned during execution. BA-POMDPs are a Bayesian RL approach that, in principle, allows for an optimal trade-off between exploitation and exploration. Unfortunately, BA-POMDPs are currently impractical to solve for any non-trivial domain. In this paper, we extend the Monte-Carlo Tree Search method POMCP to BA-POMDPs and show that the resulting method, which we call BA-POMCP, is able to tackle problems that previous solution methods have been unable to solve. Additionally, we introduce several techniques that exploit the BA-POMDP structure to improve the efficiency of BA-POMCP along with proof of their convergence.

Motivation & Objective

• Address the impracticality of solving Bayes-Adaptive POMDPs (BA-POMDPs) in non-trivial domains due to their countably infinite state space.
• Extend the POMCP algorithm to handle the Bayesian learning aspect of BA-POMDPs, enabling online planning with model uncertainty.
• Improve sample efficiency and scalability by exploiting the structural properties of BA-POMDPs, such as Dirichlet priors and sufficient statistics.
• Provide theoretical convergence guarantees for the proposed enhancements to ensure correctness in learning.
• Enable practical deployment of Bayesian RL in complex, partially observable environments where full model knowledge is unavailable.

Proposed method

• Adapt POMCP’s online, sample-based planning to the BA-POMDP framework by maintaining a belief over possible environment models.
• Introduce root sampling to sample initial states from the agent’s belief, improving exploration in the model space.
• Implement expected models sampling to approximate the expected return over model distributions, reducing variance in value estimation.
• Propose linking states to compress and efficiently represent the model space, reducing computational overhead.
• Leverage conjugate priors (Dirichlet distributions) to analytically update model beliefs using sufficient statistics, enabling efficient Bayesian updating.
• Prove convergence of the modified algorithm to the true BA-POMDP solution under standard MCTS assumptions, ensuring theoretical soundness.

Experimental results

Research questions

• RQ1 Can Monte Carlo Tree Search be effectively extended to solve Bayes-Adaptive POMDPs, which involve learning the environment model during execution?
• RQ2 How can the computational complexity of BA-POMDPs—due to their infinite model space—be reduced while preserving optimality guarantees?
• RQ3 What sampling and representation techniques can improve sample efficiency and scalability in online Bayesian POMDP planning?
• RQ4 Does the proposed method, BA-POMCP, converge to the optimal policy in the BA-POMDP framework?
• RQ5 Can structural properties of the BA-POMDP (e.g., Dirichlet conjugacy) be exploited to design more efficient inference and planning algorithms?

Key findings

• BA-POMCP successfully solves BA-POMDP problems that were previously intractable with existing methods, demonstrating scalability to larger domains.
• The integration of root sampling and expected models significantly improves sample efficiency and reduces variance in value estimation.
• Linking states reduce the dimensionality of the model representation, leading to faster belief updates and reduced memory usage.
• Theoretical convergence proofs confirm that BA-POMCP converges to the optimal solution of the underlying BA-POMDP as the number of simulations increases.
• Empirical results show that BA-POMCP outperforms baseline methods in terms of cumulative reward and learning speed in partially observable, model-unknown environments.
• The use of conjugate priors (Dirichlet) enables exact Bayesian updating of model beliefs, improving accuracy and stability during learning.