[Paper Review] Rollout Sampling Policy Iteration for Decentralized POMDPs

Feng Wu, Shlomo Zilberstein+1 more|arXiv (Cornell University)|Mar 15, 2012

Auction Theory and ApplicationsDecision Sciences21 references21 citations

TL;DR

This paper introduces DecRSPI, a rollout sampling policy iteration algorithm for decentralized partially observable Markov decision processes (DEC-POMDPs). It uses Monte Carlo sampling to generate reachable belief states and computes joint policies via rollout estimation, enabling linear time complexity in the number of agents, bounded memory use, and high-quality solutions for large-scale, model-free multi-agent problems.

ABSTRACT

We present decentralized rollout sampling policy iteration (DecRSPI) - a new algorithm for multi-agent decision problems formalized as DEC-POMDPs. DecRSPI is designed to improve scalability and tackle problems that lack an explicit model. The algorithm uses Monte- Carlo methods to generate a sample of reachable belief states. Then it computes a joint policy for each belief state based on the rollout estimations. A new policy representation allows us to represent solutions compactly. The key benefits of the algorithm are its linear time complexity over the number of agents, its bounded memory usage and good solution quality. It can solve larger problems that are intractable for existing planning algorithms. Experimental results confirm the effectiveness and scalability of the approach.

Motivation & Objective

• Address the scalability limitations of existing planning algorithms for decentralized POMDPs (DEC-POMDPs).
• Tackle multi-agent decision problems without requiring an explicit model of the environment.
• Enable efficient policy learning in large-scale DEC-POMDPs where traditional methods fail due to computational intractability.
• Develop a compact policy representation that supports scalable computation and memory efficiency.
• Improve solution quality and scalability for problems that are intractable with prior approaches.

Proposed method

• Uses Monte Carlo sampling to generate a set of reachable belief states from the initial belief distribution.
• Applies rollout estimation to evaluate and improve joint policies for each sampled belief state.
• Employs a new policy representation that enables compact encoding of joint policies across agents.
• Implements policy iteration using sampled belief states to iteratively refine the joint policy.
• Leverages rollout sampling to approximate value functions without requiring full model knowledge.
• Maintains bounded memory usage by focusing on sampled belief states rather than full belief space enumeration.

Experimental results

Research questions

• RQ1 Can rollout sampling policy iteration be effectively adapted to decentralized POMDPs with no explicit model?
• RQ2 Does the proposed method achieve linear time complexity with respect to the number of agents?
• RQ3 Can DecRSPI solve larger DEC-POMDP problems that are intractable for existing algorithms?
• RQ4 How does the compact policy representation impact memory usage and solution quality?
• RQ5 What is the trade-off between computational efficiency and solution quality in model-free multi-agent planning?

Key findings

• DecRSPI achieves linear time complexity with respect to the number of agents, significantly improving scalability.
• The algorithm uses bounded memory, making it suitable for large-scale problems where memory is constrained.
• Experimental results show that DecRSPI outperforms existing algorithms on benchmark DEC-POMDP problems, including those previously intractable.
• The method produces high-quality solutions even in the absence of a complete model, demonstrating effectiveness in model-free settings.
• The compact policy representation enables efficient storage and computation without sacrificing solution quality.
• Rollout sampling provides a robust approximation of value functions, leading to improved policy convergence and performance.