[Paper Review] Any-Play: An Intrinsic Augmentation for Zero-Shot Coordination

Keane Lucas, Ross Allen|arXiv (Cornell University)|Jan 28, 2022

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper introduces Any-Play, an intrinsic reward-based augmentation that enables self-play-trained agents to generalize to zero-shot coordination with diverse, non-identical agents in inter-algorithm cross-play settings. By promoting behavioral diversity during training, Any-Play significantly improves performance in Hanabi, achieving state-of-the-art results in zero-shot coordination with previously unmatched generalization across algorithmically dissimilar agents.

ABSTRACT

Cooperative artificial intelligence with human or superhuman proficiency in collaborative tasks stands at the frontier of machine learning research. Prior work has tended to evaluate cooperative AI performance under the restrictive paradigms of self-play (teams composed of agents trained together) and cross-play (teams of agents trained independently but using the same algorithm). Recent work has indicated that AI optimized for these narrow settings may make for undesirable collaborators in the real-world. We formalize an alternative criteria for evaluating cooperative AI, referred to as inter-algorithm cross-play, where agents are evaluated on teaming performance with all other agents within an experiment pool with no assumption of algorithmic similarities between agents. We show that existing state-of-the-art cooperative AI algorithms, such as Other-Play and Off-Belief Learning, under-perform in this paradigm. We propose the Any-Play learning augmentation -- a multi-agent extension of diversity-based intrinsic rewards for zero-shot coordination (ZSC) -- for generalizing self-play-based algorithms to the inter-algorithm cross-play setting. We apply the Any-Play learning augmentation to the Simplified Action Decoder (SAD) and demonstrate state-of-the-art performance in the collaborative card game Hanabi.

Motivation & Objective

• To address the limitation of existing cooperative AI methods that underperform when collaborating with agents trained with different algorithms.
• To formalize a new evaluation paradigm—inter-algorithm cross-play—where agents are tested with all other agents in a pool regardless of algorithmic similarity.
• To develop a general-purpose augmentation that enables self-play-based agents to achieve strong zero-shot coordination performance in this broader setting.
• To demonstrate the effectiveness of the method in a challenging cooperative environment: the card game Hanabi.

Proposed method

• The Any-Play method introduces a multi-agent intrinsic reward that encourages agents to explore diverse behaviors during self-play training.
• The intrinsic reward is computed based on the behavioral diversity of agents in the team, measured via embedding similarity of their actions or policies.
• The reward is integrated into the training objective of a self-play algorithm, such as Simplified Action Decoder (SAD), to promote exploration of non-redundant strategies.
• The method enables agents to learn policies that are robust and generalizable to teammates trained with different algorithms.
• It extends existing intrinsic reward frameworks to the multi-agent setting, focusing on inter-agent coordination diversity.
• The approach does not require architectural changes to the base algorithm, making it a plug-in augmentation for existing self-play methods.

Experimental results

Research questions

• RQ1 Can self-play-trained agents achieve strong zero-shot coordination performance when paired with agents trained using different algorithms?
• RQ2 How does behavioral diversity during training affect generalization to unseen teammates in inter-algorithm cross-play?
• RQ3 To what extent does the Any-Play intrinsic reward improve coordination performance compared to standard self-play and other intrinsic reward baselines?
• RQ4 Does Any-Play enable state-of-the-art performance in zero-shot coordination tasks like Hanabi?
• RQ5 Can the method be applied effectively to existing self-play algorithms without architectural modifications?

Key findings

• Any-Play significantly improves zero-shot coordination performance in inter-algorithm cross-play, outperforming baseline methods like Other-Play and Off-Belief Learning.
• When applied to the Simplified Action Decoder (SAD), Any-Play achieves state-of-the-art performance in Hanabi under zero-shot coordination settings.
• Agents trained with Any-Play demonstrate strong generalization to teammates trained with different algorithms, even when those algorithms are not explicitly modeled during training.
• The intrinsic diversity reward leads to more effective and varied coordination strategies, as evidenced by higher team scores across diverse teammate combinations.
• The method maintains performance gains without requiring changes to the base learning algorithm, confirming its modularity and broad applicability.
• Empirical results show that Any-Play enables SAD to achieve near-optimal performance in Hanabi when paired with diverse teammates, validating its effectiveness in real-world collaborative scenarios.