[Paper Review] Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games

Peng Peng, Ying Wen+5 more|arXiv (Cornell University)|Mar 29, 2017

Reinforcement Learning in RoboticsComputer Science40 references272 citations

TL;DR

BiCNet enables scalable multiagent coordination in StarCraft combat tasks via bidirectional recurrent communication and a shared actor-critic framework, achieving human-level coordination without demonstrations.

ABSTRACT

Many artificial intelligence (AI) applications often require multiple intelligent agents to work in a collaborative effort. Efficient learning for intra-agent communication and coordination is an indispensable step towards general AI. In this paper, we take StarCraft combat game as a case study, where the task is to coordinate multiple agents as a team to defeat their enemies. To maintain a scalable yet effective communication protocol, we introduce a Multiagent Bidirectionally-Coordinated Network (BiCNet ['bIknet]) with a vectorised extension of actor-critic formulation. We show that BiCNet can handle different types of combats with arbitrary numbers of AI agents for both sides. Our analysis demonstrates that without any supervisions such as human demonstrations or labelled data, BiCNet could learn various types of advanced coordination strategies that have been commonly used by experienced game players. In our experiments, we evaluate our approach against multiple baselines under different scenarios; it shows state-of-the-art performance, and possesses potential values for large-scale real-world applications.

Motivation & Objective

• Motivate the study of learning human-like coordination among multiple agents in complex, real-time games.
• Develop a scalable communication mechanism that handles arbitrary numbers of agents without human demonstrations.
• Demonstrate that bidirectional communication embedded in memory-augmented networks enables emergent coordination strategies.
• Show improved performance over baselines in diverse StarCraft combat scenarios.
• Illustrate the potential of BiCNet for real-world large-scale multiagent applications.

Proposed method

• Formulate StarCraft micromanagement as a zero-sum stochastic game with homogeneous agents on each side.
• Propose BiCNet: a bidirectionally coordinated multiagent actor-critic network using bi-directional RNNs for inter-agent communication and shared parameters.
• Derive a multiagent deterministic policy gradient framework (the Multiagent Deterministic PG Theorem) for learning the joint policy.
• Introduce local reward modeling to capture individual agent attribution and facilitate scalable credit assignment.
• Train with off-policy deterministic actor-critic methods and use gradients propagated through the BiCNet to update both actor and critic networks.

Experimental results

Research questions

• RQ1 Can BiCNet facilitate scalable, emergent coordination among many agents without human demonstrations?
• RQ2 Does bidirectional, memory-based communication improve multiagent coordination and performance in StarCraft combat tasks compared to existing baselines?
• RQ3 How does BiCNet handle variable team sizes and heterogeneous agent types while maintaining compact parameterization?
• RQ4 What coordination strategies emerge as agents learn, and how do these compare to human strategies (e.g., focus fire, cover attacks)?

Key findings

• BiCNet achieves state-of-the-art performance across multiple StarCraft combat scenarios relative to rule-based and several deep RL baselines.
• The model learns diverse coordination strategies such as collision-free movement, hit-and-run, cover attacks, and coordinated focus fire.
• BiCNet scales to different numbers of agents due to parameter sharing, with performance gains especially noticeable as team size increases.
• Visualization indicates high-Q states correspond to strategically advantageous coordinated actions.
• A simpler 3-agent communication example demonstrates effective bidirectional information sharing, outperforming other baselines in additive tasks.