[Paper Review] Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning

Arrasy Rahman, Niklas Höpner+2 more|arXiv (Cornell University)|Jun 18, 2020

Mobile Crowdsensing and Crowdsourcing34 references4 citations

TL;DR

This paper proposes Graph-based Policy Learning (GPL), a novel method for open ad hoc teamwork that enables a single agent to dynamically adapt to changing team compositions with agents of unknown policies. Using graph neural networks to model variable-sized teams and integrating agent models with joint-action value functions, GPL outperforms baselines in multi-environment benchmarks, especially in generalizing to unseen team sizes and compositions.

ABSTRACT

Ad hoc teamwork is the challenging problem of designing an autonomous agent which can adapt quickly to collaborate with teammates without prior coordination mechanisms, including joint training. Prior work in this area has focused on closed teams in which the number of agents is fixed. In this work, we consider open teams by allowing agents with different fixed policies to enter and leave the environment without prior notification. Our solution builds on graph neural networks to learn agent models and joint-action value models under varying team compositions. We contribute a novel action-value computation that integrates the agent model and joint-action value model to produce action-value estimates. We empirically demonstrate that our approach successfully models the effects other agents have on the learner, leading to policies that robustly adapt to dynamic team compositions and significantly outperform several alternative methods.

Motivation & Objective

• To address the challenge of open ad hoc teamwork, where agents with unknown policies dynamically join or leave a team without prior coordination.
• To overcome the limitations of fixed-state-vector RL methods in variable-sized team environments.
• To enable a single agent to learn robust policies that adapt to diverse teammate types and changing team compositions.
• To develop a joint-action value model that disentangles the impact of individual agents on the learner’s returns.
• To improve generalization to unseen team sizes and compositions through graph-based modeling and integrated action-value computation.

Proposed method

• GPL uses graph neural networks (GNNs) to model both the agent dynamics and joint-action value functions, enabling handling of variable team sizes.
• It introduces a novel action-value computation that fuses predictions from an agent model with joint-action value estimates from a GNN-based value model.
• The agent model predicts teammate behaviors using observation histories, while the joint-action value model estimates returns based on team-wide actions.
• GPL is compatible with value-based single-agent RL algorithms, with two variants tested: one based on Q-learning and one on soft policy iteration.
• The joint-action value model is trained to disentangle the contribution of each agent’s action to the learner’s return, enabling robust adaptation.
• The framework uses a pairwise utility model (MLPδ) to capture inter-agent dependencies and improve value estimation.

Experimental results

Research questions

• RQ1 Can a single-agent reinforcement learning agent effectively adapt to dynamic team compositions with unknown policies in open environments?
• RQ2 How does graph-based modeling improve performance and generalization in open ad hoc teamwork compared to fixed-size or concatenation-based approaches?
• RQ3 To what extent can GPL disentangle the effects of individual teammates on the learner’s returns, even when team size and composition vary?
• RQ4 Does integrating agent models with joint-action value functions lead to better policy learning than standalone models or input concatenation?
• RQ5 How well does GPL generalize to previously unseen team sizes and teammate types in complex multi-agent environments?

Key findings

• GPL significantly outperformed all baselines, including Q-learning, MADDPG, DGN, and ablated versions using input concatenation, across all three benchmark environments: Level-based Foraging, Wolfpack, and FortAttack.
• Both GPL variants achieved higher returns than all baselines in most learning tasks, with the GNN-based version showing superior performance in dynamic team scenarios.
• GPL demonstrated strong generalization to previously unseen team sizes and compositions, outperforming methods that failed to model dynamic team dynamics.
• The analysis of state values revealed that GPL learners learned to assign higher utility to states where attackers were within shooting range, unlike baselines that preferred states with no attackers.
• The GNN-based joint-action value model enabled better distinction of critical game states, such as the presence of nearby attackers, which correlated with improved policy performance.
• The integration of agent models with joint-action value functions allowed GPL to learn complex inter-agent dependencies that single-agent baselines failed to capture.