[Paper Review] Locally Interdependent Multi-Agent MDP: Theoretical Framework for Decentralized Agents with Dynamic Dependencies

Alex DeWeese, Guannan Qu|arXiv (Cornell University)|Jun 10, 2024

Multi-Agent Systems and Negotiation4 citations

TL;DR

This paper introduces the Locally Interdependent Multi-Agent MDP, a theoretical framework for decentralized multi-agent systems with dynamic, proximity-based dependencies. It proposes three closed-form policies—Amalgam, Cutoff, and First Step Finite Horizon Optimal—that achieve near-optimal performance, with the partially observable decentralized solution exponentially close to the fully observable optimal solution as visibility radius increases.

ABSTRACT

Many multi-agent systems in practice are decentralized and have dynamically varying dependencies. There has been a lack of attempts in the literature to analyze these systems theoretically. In this paper, we propose and theoretically analyze a decentralized model with dynamically varying dependencies called the Locally Interdependent Multi-Agent MDP. This model can represent problems in many disparate domains such as cooperative navigation, obstacle avoidance, and formation control. Despite the intractability that general partially observable multi-agent systems suffer from, we propose three closed-form policies that are theoretically near-optimal in this setting and can be scalable to compute and store. Consequentially, we reveal a fundamental property of Locally Interdependent Multi-Agent MDP's that the partially observable decentralized solution is exponentially close to the fully observable solution with respect to the visibility radius. We then discuss extensions of our closed-form policies to further improve tractability. We conclude by providing simulations to investigate some long horizon behaviors of our closed-form policies.

Motivation & Objective

• Address the lack of theoretical analysis for decentralized multi-agent systems with dynamically varying dependencies.
• Model real-world applications such as cooperative navigation, obstacle avoidance, and formation control using a metric-space-based MDP with time-varying dependence and communication graphs.
• Develop theoretically grounded, scalable policies for partially observable decentralized settings where agents have limited visibility and interaction ranges.
• Establish a fundamental theoretical property: the performance gap between decentralized and fully observable optimal policies decays exponentially with visibility radius.
• Provide a provably near-optimal, closed-form solution framework that is both computationally tractable and practically implementable.

Proposed method

• Propose the Locally Interdependent Multi-Agent MDP model, where agents interact only within a radius $\mathcal{R}$ and communicate within a visibility radius $\mathcal{V}$, both dynamically changing over time.
• Define three closed-form policies: Amalgam, Cutoff, and First Step Finite Horizon Optimal, each designed to maximize expected cumulative reward under partial observability.
• Use a telescoping lemma and reward adjustment technique to bound the performance gap between the proposed policy and the optimal centralized policy.
• Construct a deterministic lower-bound MDP with collision penalties to prove that no decentralized policy can outperform the derived upper bounds by more than a constant factor.
• Apply dynamic programming and value function decomposition to derive theoretical performance guarantees under partial observability.
• Extend the framework to improve scalability through policy modifications and validate long-horizon behaviors via simulations in obstacle avoidance, navigation, and formation control.

Figure 1: 3 agents moving in the space of $\mathcal{X}=\mathbb{R}^{2}$ with standard Euclidean distance. The bottom two agents potentially have an interdependent reward since they are within distance $\mathcal{R}$ of one another. Furthermore, every agent is within distance $\mathcal{V}$ of another a

Experimental results

Research questions

• RQ1 Can we theoretically model decentralized multi-agent systems with dynamically varying local dependencies and partial observability?
• RQ2 Are there closed-form, scalable policies that achieve near-optimal performance in such decentralized, partially observable settings?
• RQ3 How does the visibility radius affect the performance gap between decentralized and fully observable optimal policies?
• RQ4 Can we establish theoretical performance bounds that are both tight and computationally tractable for decentralized multi-agent decision-making?
• RQ5 How do the proposed policies behave over long horizons in practical multi-agent tasks like navigation and formation control?

Key findings

• The Amalgam, Cutoff, and First Step Finite Horizon Optimal policies achieve theoretical performance guarantees that are within a constant factor of the optimal centralized solution.
• The performance of the decentralized policy is exponentially close to the fully observable optimal policy, with the gap bounded by $\frac{2}{1-\gamma}\gamma^{c+1}\tilde{r}$, where $c = \lfloor(\mathcal{V} - \mathcal{R})/2\rfloor$.
• A lower bound construction proves that no decentralized policy can achieve better performance than the derived upper bounds by more than a constant factor, confirming near-optimality.
• The visibility radius $\mathcal{V}$ controls the exponential decay of the performance gap: larger visibility leads to significantly better decentralized performance.
• Simulations in obstacle avoidance, cooperative navigation, and formation control demonstrate stable, long-horizon behaviors consistent with theoretical expectations.
• Extensions to the policies improve scalability without sacrificing theoretical performance guarantees, enabling practical deployment in complex systems.

Figure 2: Bullseye Problem: In red is the optimal policy with a discounted sum of rewards of $8.85$ . The top three in blue are Amalgam Policy rollouts with $\mathcal{V}=25$ , $\mathcal{\mathcal{missing}}V=35$ , $\mathcal{V}=45$ top to bottom. They have a total discounted reward of $6.74$ , $8.26$ ,