[Paper Review] Optimizing Memory-Bounded Controllers for Decentralized POMDPs

Christopher Amato, Daniel S. Bernstein+1 more|arXiv (Cornell University)|Jun 20, 2012

Distributed Control Multi-Agent Systems10 references48 citations

TL;DR

This paper proposes a nonlinear optimization framework for learning memory-bounded stochastic finite-state controllers in infinite-horizon decentralized POMDPs. By formulating policy optimization as a nonlinear program and incorporating a correlation device, the approach achieves higher-quality solutions than state-of-the-art methods with only modest increases in memory and computation.

ABSTRACT

We present a memory-bounded optimization approach for solving infinite-horizon decentralized POMDPs. Policies for each agent are represented by stochastic finite state controllers. We formulate the problem of optimizing these policies as a nonlinear program, leveraging powerful existing nonlinear optimization techniques for solving the problem. While existing solvers only guarantee locally optimal solutions, we show that our formulation produces higher quality controllers than the state-of-the-art approach. We also incorporate a shared source of randomness in the form of a correlation device to further increase solution quality with only a limited increase in space and time. Our experimental results show that nonlinear optimization can be used to provide high quality, concise solutions to decentralized decision problems under uncertainty.

Motivation & Objective

• Address the challenge of solving infinite-horizon decentralized POMDPs with limited memory resources.
• Develop a scalable method for learning high-quality policies in decentralized, partially observable environments.
• Improve solution quality over existing approaches by leveraging nonlinear optimization techniques.
• Introduce a correlation device to enhance coordination among agents without significant computational overhead.
• Enable concise, compact controllers that balance performance and memory usage.

Proposed method

• Represent each agent's policy as a stochastic finite-state controller with a fixed number of internal states.
• Formulate the policy optimization problem as a nonlinear program over the controller's parameters.
• Use off-the-shelf nonlinear optimization solvers to find locally optimal controller parameters.
• Incorporate a shared correlation device to coordinate actions across agents, improving joint performance.
• Balance solution quality and memory cost by constraining the number of states in each controller.
• Leverage existing nonlinear programming techniques to efficiently search the policy space.

Experimental results

Research questions

• RQ1 Can nonlinear programming effectively optimize memory-bounded controllers in decentralized POMDPs?
• RQ2 How does the inclusion of a correlation device affect solution quality and computational cost?
• RQ3 To what extent does this approach outperform existing state-of-the-art methods in terms of performance and compactness?
• RQ4 How scalable is the method to larger or more complex decentralized decision problems?
• RQ5 What trade-offs exist between controller size, solution quality, and computational requirements?

Key findings

• The proposed nonlinear programming formulation produces higher-quality controllers than the state-of-the-art approach on benchmark problems.
• Incorporating a correlation device significantly improves joint performance with only a limited increase in memory and time complexity.
• The method generates concise, memory-efficient controllers that maintain strong performance in infinite-horizon decentralized POMDPs.
• Nonlinear optimization techniques are effective for searching the policy space of decentralized POMDPs despite local optimality guarantees.
• The approach demonstrates scalability and practicality for real-world decentralized decision-making under uncertainty.