[Paper Review] GCS: Graph-based Coordination Strategy for Multi-Agent Reinforcement Learning

Jingqing Ruan, Yali Du+7 more|arXiv (Cornell University)|Jan 17, 2022

Evacuation and Crowd Dynamics25 citations

TL;DR

GCS factorizes a joint MARL policy into a graph generator that produces DAG-based action coordination and a graph-based coordinated policy, enabling structured coordination among agents and improved performance across MARL benchmarks.

ABSTRACT

Many real-world scenarios involve a team of agents that have to coordinate their policies to achieve a shared goal. Previous studies mainly focus on decentralized control to maximize a common reward and barely consider the coordination among control policies, which is critical in dynamic and complicated environments. In this work, we propose factorizing the joint team policy into a graph generator and graph-based coordinated policy to enable coordinated behaviours among agents. The graph generator adopts an encoder-decoder framework that outputs directed acyclic graphs (DAGs) to capture the underlying dynamic decision structure. We also apply the DAGness-constrained and DAG depth-constrained optimization in the graph generator to balance efficiency and performance. The graph-based coordinated policy exploits the generated decision structure. The graph generator and coordinated policy are trained simultaneously to maximize the discounted return. Empirical evaluations on Collaborative Gaussian Squeeze, Cooperative Navigation, and Google Research Football demonstrate the superiority of the proposed method.

Motivation & Objective

• Motivate the need for coordinated policies in multi-agent systems beyond fully decentralized control.
• Propose a framework that factorizes the joint policy into a graph generator and a graph-based coordinated policy.
• Learn directed acyclic graphs (DAGs) to represent action dependencies among agents.
• Balance efficiency and performance via DAGness and DAG depth constraints in the graph generator.
• Train the graph generator and coordinated policy jointly to maximize discounted return.

Proposed method

• Introduce an action coordination graph (ACG) represented as a DAG whose nodes are agents and edges denote action dependencies.
• Factor the joint policy as pi(u|s,A)=rho(A|s) * prod_i pi^i(u^i|o^i,u^{pa(i)~A}) where A is generated by a DAG-based graph generator rho.
• Use a DAGness constraint g(A)=trace(exp(A∘A))−d=0 to enforce acyclicity and a DAG depth constraint via nilpotent matrices to bound hierarchy.
• Optimize the graph generator rho and the graph-based coordinated policies pi^i simultaneously to maximize the expected discounted return eta.
• Derive gradients for policy and graph generator, and apply augmented Lagrangian penalties to enforce DAG constraints (Equations 4–11).
• Describe implementation with a GAT-based encoder and MLP decoder for graph generation, and an RNN-based actor-critic for the coordinated policy.

Experimental results

Research questions

• RQ1 Can a graph-based coordination structure (ACG) capture the underlying decision dependencies among agents in MARL?
• RQ2 Does jointly training a DAG-generating graph and a graph-based coordinated policy improve coordination and returns across MARL benchmarks?
• RQ3 How do DAGness and DAG depth constraints affect learning efficiency and performance?
• RQ4 Is the DAG-based coordination robust and scalable across diverse environments like CGS, CN, and Google Football?

Key findings

• GCS achieves superior performance over baselines (VDN, QMIX, DCG, DGN) on Collaborative Gaussian Squeeze, Cooperative Navigation, and Google Football.
• The learned ACG reveals meaningful action dependencies and hierarchical decision order among agents, improving coordination.
• Increasing DAG depth improves performance up to a point but increases training time and may eventually degrade performance; an optimal depth (k) balances efficiency and gains.
• The method demonstrates faster convergence and lower variance in cooperative tasks compared to baselines that ignore explicit action dependencies.
• GRF experiments show GCS achieves higher rewards across 3-vs-2, 3-vs-6, and 5-vs-5 scenarios, illustrating robustness to dynamics and stochasticity.