[Paper Review] Influence-Optimistic Local Values for Multiagent Planning --- Extended Version

Frans A. Oliehoek, Matthijs T. J. Spaan+1 more|arXiv (Cornell University)|Feb 18, 2015

Multi-Agent Systems and Negotiation48 references3 citations

TL;DR

This paper introduces influence-optimistic upper bounds (IO-UBs) for factored Dec-POMDPs without factored value functions, enabling tight global upper bounds via local sub-problem decomposition with optimistic assumptions about external influences. The method achieves empirical approximation factors below 1.7 for heuristic solutions in problems with hundreds of agents, providing strong quality guarantees for large-scale multiagent planning.

ABSTRACT

Recent years have seen the development of methods for multiagent planning under uncertainty that scale to tens or even hundreds of agents. However, most of these methods either make restrictive assumptions on the problem domain, or provide approximate solutions without any guarantees on quality. Methods in the former category typically build on heuristic search using upper bounds on the value function. Unfortunately, no techniques exist to compute such upper bounds for problems with non-factored value functions. To allow for meaningful benchmarking through measurable quality guarantees on a very general class of problems, this paper introduces a family of influence-optimistic upper bounds for factored decentralized partially observable Markov decision processes (Dec-POMDPs) that do not have factored value functions. Intuitively, we derive bounds on very large multiagent planning problems by subdividing them in sub-problems, and at each of these sub-problems making optimistic assumptions with respect to the influence that will be exerted by the rest of the system. We numerically compare the different upper bounds and demonstrate how we can achieve a non-trivial guarantee that a heuristic solution for problems with hundreds of agents is close to optimal. Furthermore, we provide evidence that the upper bounds may improve the effectiveness of heuristic influence search, and discuss further potential applications to multiagent planning.

Motivation & Objective

• To address the lack of scalable, guaranteed upper bounds for large-scale multiagent planning under uncertainty.
• To enable performance guarantees for heuristic solutions in factored Dec-POMDPs that do not admit factored value functions.
• To develop a general technique for computing upper bounds in non-factored value function settings using local optimistic assumptions about system-wide influences.
• To demonstrate the utility of these bounds in benchmarking heuristic methods and improving heuristic search effectiveness.
• To support practical deployment and theoretical analysis of multiagent systems by quantifying solution quality gaps.

Proposed method

• Decomposes large factored Dec-POMDPs into smaller sub-problems based on a partition of state factors and agent interactions.
• Computes local upper bounds by assuming optimistic influence from the rest of the system on each sub-problem, enabling independent solution of sub-problems.
• Uses influence-optimism to decouple sub-problems: assumes the most favorable possible influence from external agents and state factors.
• Combines local upper bounds into a global upper bound on the full problem’s value, leveraging the structure of the factored model.
• Employs a partition-based decomposition to manage scalability, ensuring sub-problems remain tractable while preserving global bound quality.
• Applies the method to compute upper bounds for heuristic solutions, enabling empirical evaluation of approximation quality.

Experimental results

Research questions

• RQ1 Can tight upper bounds be computed for large-scale factored Dec-POMDPs that do not have factored value functions?
• RQ2 Can influence-optimistic assumptions on external influences yield effective and scalable upper bounds for multiagent planning?
• RQ3 To what extent can these upper bounds provide meaningful quality guarantees for heuristic solutions in problems with hundreds of agents?
• RQ4 How do influence-optimistic upper bounds compare to existing methods in terms of tightness and computational feasibility?
• RQ5 Can these bounds enhance the effectiveness of heuristic search methods in multiagent planning?

Key findings

• The proposed influence-optimistic upper bounds achieve empirical approximation factors of less than 1.7 for heuristic solutions in factored Dec-POMDPs with hundreds of agents.
• The method provides non-trivial, measurable quality guarantees for heuristic solutions in problems where no such guarantees were previously available.
• Empirical results show that the upper bounds are tight enough to meaningfully assess the performance gap of heuristic methods in large-scale multiagent systems.
• The bounds improve the effectiveness of heuristic influence search, as evidenced by experiments showing significant performance gains.
• The approach generalizes prior methods that rely on value factorization, extending upper bound computation to non-factored value functions.
• Evidence suggests that influence strength is a key dimension of weak coupling, influencing how well localized approximations perform.