[Paper Review] A game-theoretic model and best-response learning method for ad hoc coordination in multiagent systems

Stefano V. Albrecht, Subramanian Ramamoorthy|arXiv (Cornell University)|May 6, 2013

Reinforcement Learning in Robotics6 references69 citations

TL;DR

This paper proposes Harsanyi-Bellman Ad Hoc Coordination (HBA), a game-theoretic framework that models multiagent ad hoc coordination as a stochastic Bayesian game using user-defined types to represent agent behaviors. HBA achieves superior performance in both simulated foraging and human-machine experiments, outperforming alternatives in welfare and winning rate with minimal type definitions.

ABSTRACT

The ad hoc coordination problem is to design an ad hoc agent which is able to achieve optimal flexibility and efficiency in a multiagent system that admits no prior coordination between the ad hoc agent and the other agents. We conceptualise this problem formally as a stochastic Bayesian game in which the behaviour of a player is determined by its type. Based on this model, we derive a solution, called Harsanyi-Bellman Ad Hoc Coordination (HBA), which utilises a set of user-defined types to characterise players based on their observed behaviours. We evaluate HBA in the level-based foraging domain, showing that it outperforms several alternative algorithms using just a few user-defined types. We also report on a human-machine experiment in which the humans played Prisoner's Dilemma and Rock-Paper-Scissors against HBA and alternative algorithms. The results show that HBA achieved equal efficiency but a significantly higher welfare and winning rate.

Motivation & Objective

• Address the ad hoc coordination problem in multiagent systems where prior coordination is impossible.
• Formalize agent behavior using a stochastic Bayesian game model with types based on observed actions.
• Design a scalable learning method that enables an ad hoc agent to adapt to diverse, uncoordinated agents.
• Evaluate HBA in both synthetic environments and human-in-the-loop experiments to validate robustness and performance.
• Demonstrate that a small set of user-defined types can yield high coordination efficiency and welfare in complex settings.

Proposed method

• Model the ad hoc coordination problem as a stochastic Bayesian game where agent types represent behavioral strategies.
• Define a set of user-specified types to characterize observed behaviors of other agents, enabling type inference.
• Apply the Harsanyi-Belief update to estimate the posterior probability of each agent type based on observations.
• Use a best-response learning mechanism to select actions that maximize expected utility given the estimated types.
• Integrate HBA into a decision-making loop that continuously updates beliefs and adapts strategies in real time.
• Leverage the Bellman optimality equation to compute optimal action sequences under uncertainty about other agents’ types.

Experimental results

Research questions

• RQ1 Can a minimal set of user-defined types effectively model diverse agent behaviors in ad hoc coordination?
• RQ2 How does HBA compare to existing algorithms in terms of coordination efficiency and welfare in structured multiagent environments?
• RQ3 To what extent can HBA achieve high performance without prior coordination or knowledge of other agents?
• RQ4 How does HBA perform in human-in-the-loop settings involving strategic games like Prisoner’s Dilemma and Rock-Paper-Scissors?
• RQ5 Does HBA maintain strong performance across varying levels of behavioral diversity and uncertainty?

Key findings

• HBA outperformed several alternative algorithms in the level-based foraging domain using only a few user-defined types.
• In human-machine experiments, HBA achieved equal efficiency but significantly higher welfare and winning rates compared to baseline algorithms.
• HBA demonstrated robust performance in both Prisoner’s Dilemma and Rock-Paper-Scissors, indicating adaptability to strategic and non-cooperative settings.
• The use of a small number of user-defined types enabled effective inference of other agents’ behaviors and optimal response selection.
• HBA achieved high coordination quality even without prior coordination, confirming its suitability for real-world multiagent systems.
• The method maintained strong performance across diverse behavioral types, suggesting scalability and generalization capability.