[Paper Review] The Complexity of Decentralized Control of Markov Decision Processes

Daniel S. Bernstein, Shlomo Zilberstein+1 more|arXiv (Cornell University)|Jan 16, 2013

Optimization and Search ProblemsComputer Science34 references237 citations

TL;DR

This paper investigates decentralized control in Markov Decision Processes (MDPs) with partial observability, introducing generalized models for multi-agent planning under uncertainty. It proves that even finite-horizon problems in these models are NEXP-complete, demonstrating that decentralized planning provably requires doubly exponential time and cannot be efficiently reduced to centralized solutions using standard techniques.

ABSTRACT

Planning for distributed agents with partial state information is considered from a decision- theoretic perspective. We describe generalizations of both the MDP and POMDP models that allow for decentralized control. For even a small number of agents, the finite-horizon problems corresponding to both of our models are complete for nondeterministic exponential time. These complexity results illustrate a fundamental difference between centralized and decentralized control of Markov processes. In contrast to the MDP and POMDP problems, the problems we consider provably do not admit polynomial-time algorithms and most likely require doubly exponential time to solve in the worst case. We have thus provided mathematical evidence corresponding to the intuition that decentralized planning problems cannot easily be reduced to centralized problems and solved exactly using established techniques.

Motivation & Objective

• To formalize decentralized control of Markov Decision Processes under partial state information for multiple agents.
• To identify the computational complexity of finite-horizon planning in decentralized settings.
• To contrast the complexity of decentralized control with centralized MDPs and POMDPs.
• To provide theoretical evidence that decentralized planning cannot be efficiently reduced to centralized approaches.

Proposed method

• Proposes a generalized MDP model allowing decentralized control with partial observability across multiple agents.
• Introduces a formal framework for decentralized partially observable MDPs (Dec-POMDPs) as an extension of standard POMDPs.
• Uses complexity-theoretic analysis to classify the computational hardness of solving finite-horizon problems in this model.
• Applies results from computational complexity theory, particularly the class NEXP, to establish completeness results.
• Analyzes decision problems under uncertainty where agents act independently based on local observations.
• Demonstrates that no polynomial-time algorithm can solve these problems unless P = NEXP.

Experimental results

Research questions

• RQ1 What is the computational complexity of finite-horizon planning in decentralized MDPs with partial observability?
• RQ2 How does the complexity of decentralized control compare to that of centralized MDPs and POMDPs?
• RQ3 Can decentralized planning problems be reduced to centralized problems using existing techniques?
• RQ4 Are there inherent limits to the efficiency of algorithms solving decentralized decision-making under uncertainty?
• RQ5 Does the structure of decentralized control inherently require more than exponential time to solve?

Key findings

• Finite-horizon problems in the proposed decentralized MDP model are complete for nondeterministic exponential time (NEXP).
• The complexity of decentralized control is fundamentally higher than that of centralized MDPs and POMDPs, which are in P and PSPACE, respectively.
• The results imply that no polynomial-time algorithm can solve these problems unless P = NEXP, which is considered highly unlikely.
• The paper provides mathematical evidence that decentralized planning cannot be efficiently reduced to centralized planning using standard techniques.
• The findings confirm the intuition that decentralized decision-making under uncertainty is inherently more complex and intractable than centralized counterparts.