[Paper Review] A Framework for Sequential Planning in Multi-Agent Settings

Prashant Doshi, Piotr J. Gmytrasiewicz|arXiv (Cornell University)|Sep 9, 2011

Reinforcement Learning in Robotics48 references365 citations

TL;DR

This paper introduces Interactive POMDPs (I-POMDPs), a decision-theoretic framework for sequential planning in multi-agent systems where agents maintain beliefs over both environmental states and other agents' models, including their beliefs and preferences. By extending POMDPs to include nested, recursive beliefs, the framework enables optimal decision-making under uncertainty while preserving convergence, piecewise linearity, and convexity of value functions—offering a more expressive alternative to Nash equilibria that avoids issues of non-uniqueness and incompleteness.

ABSTRACT

This paper extends the framework of partially observable Markov decision processes (POMDPs) to multi-agent settings by incorporating the notion of agent models into the state space. Agents maintain beliefs over physical states of the environment and over models of other agents, and they use Bayesian updates to maintain their beliefs over time. The solutions map belief states to actions. Models of other agents may include their belief states and are related to agent types considered in games of incomplete information. We express the agents autonomy by postulating that their models are not directly manipulable or observable by other agents. We show that important properties of POMDPs, such as convergence of value iteration, the rate of convergence, and piece-wise linearity and convexity of the value functions carry over to our framework. Our approach complements a more traditional approach to interactive settings which uses Nash equilibria as a solution paradigm. We seek to avoid some of the drawbacks of equilibria which may be non-unique and do not capture off-equilibrium behaviors. We do so at the cost of having to represent, process and continuously revise models of other agents. Since the agents beliefs may be arbitrarily nested, the optimal solutions to decision making problems are only asymptotically computable. However, approximate belief updates and approximately optimal plans are computable. We illustrate our framework using a simple application domain, and we show examples of belief updates and value functions.

Motivation & Objective

• To develop a normative framework for sequential decision-making in multi-agent environments with uncertainty.
• To extend POMDPs by incorporating agents' beliefs about other agents' models, including their beliefs and preferences.
• To address limitations of Nash equilibria, such as non-uniqueness and incompleteness, by using a belief-based, optimal-response approach.
• To formalize interactive beliefs as nested, hierarchical constructs that are updated via Bayesian inference.
• To demonstrate that key properties of POMDPs—such as value function convexity and convergence of value iteration—carry over to the multi-agent setting.

Proposed method

• Proposes I-POMDPs as an extension of POMDPs, where the state space includes both physical states and models of other agents.
• Models agents' beliefs over their own and others' types, preferences, and beliefs, allowing for arbitrarily nested interactive beliefs.
• Uses Bayesian updating to recursively revise beliefs based on observations and actions, generalizing POMDP belief updates.
• Defines solutions as mappings from belief states to actions, with value functions computed via dynamic programming and value iteration.
• Introduces finitely nested I-POMDPs as a computable approximation to infinite nesting, enabling practical computation.
• Employs alpha vectors and inner products to represent and compute piecewise linear and convex value functions.

Experimental results

Research questions

• RQ1 How can agents maintain and update beliefs about other agents' models, including their beliefs and preferences, in a recursive, hierarchical manner?
• RQ2 Can the convergence, piecewise linearity, and convexity of value functions in POMDPs be preserved in a multi-agent setting with interactive beliefs?
• RQ3 What are the computational trade-offs of maintaining infinitely nested beliefs, and how can they be approximated effectively?
• RQ4 How does the I-POMDP framework compare to traditional POMDPs and Nash equilibrium solutions in terms of solution quality and expressiveness?
• RQ5 Under what conditions do solutions to I-POMDPs converge, and how fast do they converge?

Key findings

• The value iteration algorithm in I-POMDPs converges to a unique fixed point, as proven via the Contraction Mapping Theorem.
• The value function in finitely nested I-POMDPs is piecewise linear and convex (PWLC), generalizing a key property of POMDPs.
• The belief update in I-POMDPs is a generalization of the POMDP update, incorporating beliefs about other agents' models.
• The framework supports optimal decision-making under uncertainty by modeling agents as rational, self-interested actors with recursive beliefs.
• Approximate belief updates and approximately optimal plans are computable, even though exact solutions are asymptotically computable due to infinite nesting.
• The framework outperforms standard POMDPs in multi-agent settings by capturing off-equilibrium behaviors and enabling better prediction of others' actions.