[Paper Review] Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning

Sheng Li, Jayesh K. Gupta+3 more|arXiv (Cornell University)|Jun 19, 2020

Reinforcement Learning in Robotics50 references38 citations

TL;DR

DICG learns dynamic implicit coordination graphs via self-attention and graph convolution to balance centralized and decentralized MARL, improving coordination in predator-prey, SMAC, and traffic junction tasks.

ABSTRACT

Multi-agent reinforcement learning (MARL) requires coordination to efficiently solve certain tasks. Fully centralized control is often infeasible in such domains due to the size of joint action spaces. Coordination graph based formalization allows reasoning about the joint action based on the structure of interactions. However, they often require domain expertise in their design. This paper introduces the deep implicit coordination graph (DICG) architecture for such scenarios. DICG consists of a module for inferring the dynamic coordination graph structure which is then used by a graph neural network based module to learn to implicitly reason about the joint actions or values. DICG allows learning the tradeoff between full centralization and decentralization via standard actor-critic methods to significantly improve coordination for domains with large number of agents. We apply DICG to both centralized-training-centralized-execution and centralized-training-decentralized-execution regimes. We demonstrate that DICG solves the relative overgeneralization pathology in predatory-prey tasks as well as outperforms various MARL baselines on the challenging StarCraft II Multi-agent Challenge (SMAC) and traffic junction environments.

Motivation & Objective

• Motivate the need for better coordination in multi-agent RL where joint action spaces are large and static coordination graphs are insufficient.
• Introduce DICG that infers a dynamic coordination graph from observations and uses a graph neural network to compute joint-action values or actions.
• Enable a trade-off between centralized and decentralized execution through CTCE and CTDE regimes.
• Show that DICG mitigates relative overgeneralization and outperforms baselines on complex MARL tasks like SMAC and traffic junction.

Proposed method

• Use a self-attention module to learn an implicit, soft coordination graph with adjacency matrix M from agent embeddings.
• Apply graph convolution on M to pass messages and integrate information across agents.
• Provide two usage modes: DICG-CE for centralized-training-centralized-execution (CTCE) and DICG-DE for centralized-training-decentralized-execution (CTDE) with a centralized baseline.
• Train the entire DICG module end-to-end with standard actor-critic methods (PPO) using either joint actions or centralized baselines for advantage estimation.
• Encoder shares parameters across agents to produce embeddings e_i from observations o_i, which are used to compute attention weights μ_ij = softmax_j attention(e_i, e_j).
• Final embeddings Ê are formed after m graph-convolution layers with a residual connection to E(0).
• DICG-CE uses Ê to generate actions for agents; DICG-DE uses Ê to estimate a centralized critic baseline for advantage estimation.

Experimental results

Research questions

• RQ1 Can a dynamically learned implicit coordination graph improve multi-agent coordination without domain-specific heuristics?
• RQ2 Does integrating attention-based graph structure with GCNs mitigate relative overgeneralization in MARL tasks?
• RQ3 How does DICG perform under CTCE and CTDE regimes compared to fully centralized or decentralized baselines?
• RQ4 Are DICG embeddings more informative for predicting other agents' actions or values than raw observations?

Key findings

• DICG solves relative overgeneralization in predator-prey tasks where fully centralized or decentralized methods struggle.
• DICG outperforms baselines on StarCraft II Multi-Agent Challenge (SMAC) scenarios and traffic junction in terms of win rate and sample efficiency.
• Learned attention weights adapt to coordination demands (e.g., higher attention to farther agents as penalties increase).
• Post-DICG embeddings better predict other agents’ actions than pre-DICG embeddings, indicating successful implicit coordination reasoning.
• In SMAC, DICG-DE-LSTM achieves the highest and most stable win rates across several maps, outperforming DCG, VDN, and QMIX baselines.
• In traffic junction, DICG-DE-MLP achieves strong performance in medium and hard modes, outperforming several decentralized and some centralized baselines.