[Paper Review] Multiagent Bidirectionally-Coordinated Nets for Learning to Play StarCraft Combat Games.

Peng Peng, Quan Yuan+5 more|arXiv (Cornell University)|Mar 29, 2017

Reinforcement Learning in Robotics29 references171 citations

TL;DR

This paper proposes BiCNet, a multiagent bidirectionally-coordinated network using a vectorized actor-critic framework to enable scalable, communication-efficient coordination among AI agents in StarCraft combat games. Without human demonstrations, BiCNet learns diverse, human-like coordination strategies across varying numbers of agents and terrains, achieving state-of-the-art performance in multi-agent combat scenarios.

ABSTRACT

Real-world artificial intelligence (AI) applications often require multiple agents to work in a collaborative effort. Efficient learning for intra-agent communication and coordination is an indispensable step towards general AI. In this paper, we take StarCraft combat game as the test scenario, where the task is to coordinate multiple agents as a team to defeat their enemies. To maintain a scalable yet effective communication protocol, we introduce a multiagent bidirectionally-coordinated network (BiCNet ['bIknet]) with a vectorised extension of actor-critic formulation. We show that BiCNet can handle different types of combats under diverse terrains with arbitrary numbers of AI agents for both sides. Our analysis demonstrates that without any supervisions such as human demonstrations or labelled data, BiCNet could learn various types of coordination strategies that is similar to these of experienced game players. Moreover, BiCNet is easily adaptable to the tasks with heterogeneous agents. In our experiments, we evaluate our approach against multiple baselines under different scenarios; it shows state-of-the-art performance, and possesses potential values for large-scale real-world applications.

Motivation & Objective

• To enable efficient, scalable coordination among multiple AI agents in complex, dynamic combat environments like StarCraft.
• To develop a communication protocol that maintains effectiveness regardless of the number of agents or terrain complexity.
• To learn coordination strategies without human demonstrations or labeled data, emulating expert-level play.
• To support heterogeneous agents in collaborative combat tasks, enhancing real-world applicability.
• To achieve state-of-the-art performance in multi-agent reinforcement learning for real-time strategy games.

Proposed method

• Introduces BiCNet, a multiagent deep reinforcement learning framework with bidirectional communication between agents to enhance coordination.
• Employs a vectorized extension of the actor-critic algorithm to scale efficiently across varying numbers of agents.
• Designs a communication mechanism that allows agents to exchange information both ways, improving joint decision-making.
• Uses a centralized critic with decentralized execution to balance communication efficiency and policy optimization.
• Applies no human demonstrations or labeled data, relying solely on self-supervised learning through environment feedback.
• Supports heterogeneous agents by adapting individual policies while maintaining joint coordination through shared communication protocols.

Experimental results

Research questions

• RQ1 Can a multiagent deep reinforcement learning framework learn effective coordination strategies in StarCraft combat without human demonstrations?
• RQ2 How well does the bidirectional communication mechanism in BiCNet scale across different numbers of agents and terrain types?
• RQ3 To what extent can BiCNet emulate human-like coordination strategies observed in expert gameplay?
• RQ4 How does BiCNet perform compared to existing baselines in diverse combat scenarios with varying agent counts?
• RQ5 Can BiCNet generalize to tasks involving heterogeneous agents while maintaining high coordination efficiency?

Key findings

• BiCNet successfully learns diverse coordination strategies in StarCraft combat without any human demonstrations or labeled data.
• The framework generalizes across different numbers of agents and terrain types, maintaining high performance in varied combat scenarios.
• BiCNet achieves state-of-the-art performance compared to multiple baselines across all evaluated scenarios.
• The bidirectional communication mechanism significantly improves coordination efficiency and team-level decision-making.
• The model demonstrates strong adaptability to heterogeneous agents, enabling effective collaboration in mixed-agent teams.
• The vectorized actor-critic formulation enables scalable training and inference, supporting large-scale multi-agent applications.