[Paper Review] Bayesian Policy Optimization for Model Uncertainty

Gilwoo Lee, Brian Hou+4 more|arXiv (Cornell University)|Oct 1, 2018

Reinforcement Learning in Robotics24 references4 citations

TL;DR

This paper proposes Bayesian Policy Optimization (BPO), a batch policy optimization method that directly learns policies for continuous Bayes-Adaptive Markov Decision Processes (BAMDPs) by reasoning over belief distributions over latent model parameters. By decoupling state and belief encoding in a neural network and using a black-box Bayes filter during offline training, BPO outperforms non-belief-based robust RL methods and matches state-of-the-art POMDP solvers on continuous control benchmarks with model uncertainty.

ABSTRACT

Addressing uncertainty is critical for autonomous systems to robustly adapt to the real world. We formulate the problem of model uncertainty as a continuous Bayes-Adaptive Markov Decision Process (BAMDP), where an agent maintains a posterior distribution over latent model parameters given a history of observations and maximizes its expected long-term reward with respect to this belief distribution. Our algorithm, Bayesian Policy Optimization, builds on recent policy optimization algorithms to learn a universal policy that navigates the exploration-exploitation trade-off to maximize the Bayesian value function. To address challenges from discretizing the continuous latent parameter space, we propose a new policy network architecture that encodes the belief distribution independently from the observable state. Our method significantly outperforms algorithms that address model uncertainty without explicitly reasoning about belief distributions and is competitive with state-of-the-art Partially Observable Markov Decision Process solvers.

Motivation & Objective

• To address model uncertainty in continuous control environments where system dynamics or rewards are partially unknown.
• To develop a scalable, end-to-end deep RL method that explicitly reasons about belief distributions over latent parameters, rather than relying on point estimates.
• To overcome the exponential growth of belief space in continuous BAMDPs through efficient belief-state representation and policy learning.
• To demonstrate that direct policy optimization over belief states can achieve performance competitive with state-of-the-art POMDP solvers while being more scalable than Monte-Carlo Tree Search or point-based value function approximation.

Proposed method

• BPO formulates the problem as a continuous BAMDP, where the agent maintains a posterior belief over latent model parameters using a black-box Bayes filter.
• The policy network takes a joint state-belief input, with separate encoders for state and belief to decouple their representations and reduce dimensionality.
• During offline training, BPO simulates the policy across multiple sampled latent models, updating beliefs along simulated trajectories without branching on actions or observations.
• The method leverages batch policy optimization (e.g., TRPO) to directly optimize the Bayesian value function, maximizing expected long-term reward under belief uncertainty.
• A belief representation is constructed via discretization of the continuous latent parameter space, with the belief vector encoded independently from the state embedding.
• The approach is agnostic to the choice of policy optimization algorithm, allowing integration with modern methods like PPO.

Experimental results

Research questions

• RQ1 Can a deep policy network be trained end-to-end to directly optimize the Bayesian value function in continuous BAMDPs without explicit value function approximation?
• RQ2 How does belief-aware policy learning compare to methods that use maximum likelihood estimates (MLE) of model parameters in the presence of model uncertainty?
• RQ3 To what extent can independent state and belief encoding mitigate the curse of dimensionality in high-dimensional belief spaces?
• RQ4 Does BPO achieve performance comparable to state-of-the-art POMDP solvers on continuous control benchmarks with latent parameter uncertainty?

Key findings

• BPO significantly outperforms EPOpt and UP-MLE on the HalfCheetah environment, achieving higher average returns across a range of MDPs.
• On the Ant environment, BPO reduces belief entropy more quickly than UP-MLE, leading to faster convergence of the belief distribution to a unimodal state.
• BPO achieves performance competitive with state-of-the-art POMDP solvers on continuous control benchmarks, despite not using value function approximation or tree search.
• The method scales well with increasing discretization of the latent parameter space, though performance degrades beyond an optimal discretization level, suggesting a need for adaptive resolution.
• In environments where beliefs collapse quickly (e.g., deterministic systems with low noise), even non-belief-based methods like UP-MLE can perform well, indicating that belief reasoning is most critical in slowly converging or highly uncertain settings.