[Paper Review] Decentralized Control of Partially Observable Markov Decision Processes using Belief Space Macro-actions

Shayegan Omidshafiei, Ali‐akbar Agha‐mohammadi+2 more|arXiv (Cornell University)|Feb 20, 2015

Bayesian Modeling and Causal Inference11 citations

TL;DR

This paper proposes a decentralized partially observable semi-Markov decision process (Dec-POSMDP) framework that enables scalable, asynchronous multi-robot planning under uncertainty by using closed-loop belief space macro-actions (MAs). The method automatically generates robust, temporally extended MAs via graph-based planning and employs a novel Monte Carlo search algorithm (MMCS) that achieves 118% higher expected policy value than standard Monte Carlo search in a complex package delivery task, enabling high-quality solutions for large-scale problems.

ABSTRACT

The focus of this paper is on solving multi-robot planning problems in continuous spaces with partial observability. Decentralized partially observable Markov decision processes (Dec-POMDPs) are general models for multi-robot coordination problems, but representing and solving Dec-POMDPs is often intractable for large problems. To allow for a high-level representation that is natural for multi-robot problems and scalable to large discrete and continuous problems, this paper extends the Dec-POMDP model to the decentralized partially observable semi-Markov decision process (Dec-POSMDP). The Dec-POSMDP formulation allows asynchronous decision-making by the robots, which is crucial in multi-robot domains. We also present an algorithm for solving this Dec-POSMDP which is much more scalable than previous methods since it can incorporate closed-loop belief space macro-actions in planning. These macro-actions are automatically constructed to produce robust solutions. The proposed method's performance is evaluated on a complex multi-robot package delivery problem under uncertainty, showing that our approach can naturally represent multi-robot problems and provide high-quality solutions for large-scale problems.

Motivation & Objective

• To address the intractability of solving large-scale Dec-POMDPs in continuous, partially observable domains with asynchronous decision-making.
• To enable scalable, high-level representation of multi-robot coordination problems using temporally extended actions (macro-actions).
• To develop a formal framework—Dec-POSMDP—that supports asynchronous, variable-duration macro-actions with analytically characterized completion times and success probabilities.
• To design an efficient search algorithm (MMCS) that leverages macro-action properties to explore the policy space effectively under computational constraints.
• To demonstrate the framework’s effectiveness on a complex, real-world multi-robot package delivery problem under uncertainty.

Proposed method

• Extends Dec-POMDP to Dec-POSMDP to model asynchronous decision-making and variable-duration macro-actions in partially observable, continuous-state domains.
• Automatically constructs closed-loop belief space macro-actions (MAs) using graph-based planning, where each LMA (local macro-action) acts as a feedback controller funneling beliefs to a terminal belief state.
• Characterizes key macro-action properties—completion time and success probability—analytically to enable integration into the Dec-POSMDP framework.
• Employs a novel Monte Carlo search algorithm (MMCS) that exploits knowledge of promising policies to improve exploration and exploitation in the policy space.
• Uses dynamic programming on the LMA graph to synthesize optimal policies for each macro-action, ensuring robustness to sensor noise and uncertainty.
• Applies the framework to a multi-robot package delivery domain with 2D continuous state spaces and stochastic observations, validating performance via simulation.

Experimental results

Research questions

• RQ1 Can macro-actions be effectively extended to decentralized, partially observable, continuous-state domains with asynchronous decision-making?
• RQ2 How can variable-duration, closed-loop macro-actions be formally characterized in terms of completion time and success probability for use in planning?
• RQ3 Can a scalable search algorithm be designed to efficiently explore the high-dimensional policy space of Dec-POSMDPs?
• RQ4 Does the use of belief space macro-actions significantly improve solution quality and scalability compared to primitive action-based Dec-POMDPs?
• RQ5 Can the framework generate high-quality, robust controllers for complex multi-robot tasks under uncertainty?

Key findings

• The MMCS algorithm achieved a 118% higher expected policy value than standard Monte Carlo search after 1000 iterations in the package delivery domain.
• After 1000 search iterations, the MMCS policy delivered up to 9 packages in some simulations, while the Monte Carlo policy had negligible success in delivering more than 2 packages.
• The success probability of delivering at least 3 packages within a fixed time horizon was significantly higher with the MMCS policy compared to Monte Carlo.
• The value function, success probability, and completion time of each macro-action were analytically characterized and available over the entire belief space, enabling robust integration into planning.
• The proposed Dec-POSMDP framework enables scalable solution of large-scale, continuous-space, partially observable multi-robot problems that are intractable with traditional Dec-POMDP methods.
• The framework supports asynchronous decision-making by modeling macro-actions with variable durations and probabilistic termination, enabling practical deployment in real-world robotic systems.