[Paper Review] Scalable Planning and Learning for Multiagent POMDPs: Extended Version

Christopher Amato, Frans A. Oliehoek|arXiv (Cornell University)|Apr 4, 2014

Reinforcement Learning in Robotics50 references3 citations

TL;DR

This paper introduces FV-POMCP, a novel online planning algorithm for multiagent POMDPs that exploits locality of interaction through factored value functions and factored statistics to scale to large action and observation spaces. By decomposing the value function into overlapping factors and using UCT-based tree search with sample-based rollouts, FV-POMCP achieves high-quality solutions in problems with up to 10 agents, significantly outperforming standard POMCP in both planning and Bayesian reinforcement learning settings.

ABSTRACT

Online, sample-based planning algorithms for POMDPs have shown great promise in scaling to problems with large state spaces, but they become intractable for large action and observation spaces. This is particularly problematic in multiagent POMDPs where the action and observation space grows exponentially with the number of agents. To combat this intractability, we propose a novel scalable approach based on sample-based planning and factored value functions that exploits structure present in many multiagent settings. This approach applies not only in the planning case, but also in the Bayesian reinforcement learning setting. Experimental results show that we are able to provide high quality solutions to large multiagent planning and learning problems.

Motivation & Objective

• To address the exponential growth in action and observation spaces that hinders scalability of existing POMDP and MPOMDP planning methods in multiagent systems.
• To develop an online, sample-based planning algorithm that efficiently handles large-scale multiagent POMDPs without requiring a fully factored model.
• To extend the applicability of sample-based planning to Bayesian reinforcement learning in multiagent settings, particularly for Bayes-Adaptive POMDPs with infinite state spaces.
• To enable efficient joint action selection and belief update in centralized multiagent systems by exploiting structural dependencies among agents.
• To demonstrate that factored value function approximation can be effectively integrated into Monte Carlo tree search for POMDPs while maintaining scalability and solution quality.

Proposed method

• FV-POMCP extends POMCP by introducing factored statistics that represent value function components based on local agent interactions, reducing the number of joint actions considered during planning.
• The algorithm uses a factored tree structure where nodes represent partial action-observation histories, enabling efficient traversal and pruning of irrelevant joint action branches.
• It applies UCT (Upper Confidence bound for Trees) to prioritize exploration in promising branches of the factored tree, balancing exploration and exploitation in belief space.
• Value function approximation is achieved by factoring the Q-value function into overlapping local components, allowing scalable computation without requiring a fully factorized MDP model.
• Root sampling is used to generate initial state samples, which are then used to simulate trajectories through the lookahead tree and perform rollouts to estimate returns.
• The method supports both planning and Bayesian reinforcement learning by treating model uncertainty as part of the belief state, enabling efficient learning in Bayes-Adaptive POMDPs.

Experimental results

Research questions

• RQ1 Can factored value function approximation be effectively integrated into online Monte Carlo tree search for multiagent POMDPs to improve scalability?
• RQ2 How does exploiting locality of interaction—where agents interact only with subsets of others—affect the performance and efficiency of sample-based planning in large MPOMDPs?
• RQ3 To what extent can FV-POMCP scale to problems with more than 4–5 agents, particularly when joint action and observation spaces grow exponentially?
• RQ4 Can FV-POMCP outperform standard POMCP in both planning and Bayesian reinforcement learning tasks for multiagent systems with large action and observation spaces?
• RQ5 Does the use of factored statistics and factored trees maintain solution quality while significantly reducing computational complexity in MPOMDPs?

Key findings

• FV-POMCP successfully scales to multiagent POMDPs with up to 10 agents, demonstrating significant improvement over standard POMCP in terms of solution quality and computational efficiency.
• The method achieves high-quality solutions in Bayesian reinforcement learning settings for MPOMDPs, including problems with infinite state spaces due to model uncertainty.
• FV-POMCP outperforms non-factored POMCP in both planning and learning tasks, confirming that the exploitation of locality of interaction leads to measurable gains.
• The use of factored statistics and factored trees reduces the number of joint actions and histories considered, enabling scalability to larger problems without sacrificing policy quality.
• Empirical results show that the algorithm maintains strong performance even when the value function is only approximately factored, making it applicable to a broad class of real-world multiagent systems.
• The approach enables efficient online planning and learning in complex multiagent environments where traditional methods fail due to exponential state, action, and observation space growth.