[Paper Review] Deep Coordination Graphs

Wendelin Böhmer, Vitaly Kurin+1 more|arXiv (Cornell University)|Sep 27, 2019

Graph Theory and Algorithms50 references42 citations

TL;DR

DCG introduces a deep reinforcement learning approach that factorizes the joint value function over a coordination graph with shared parameters and low-rank payoffs, enabling scalable, end-to-end Q-learning for multi-agent tasks.

ABSTRACT

This paper introduces the deep coordination graph (DCG) for collaborative multi-agent reinforcement learning. DCG strikes a flexible trade-off between representational capacity and generalization by factoring the joint value function of all agents according to a coordination graph into payoffs between pairs of agents. The value can be maximized by local message passing along the graph, which allows training of the value function end-to-end with Q-learning. Payoff functions are approximated with deep neural networks that employ parameter sharing and low-rank approximations to significantly improve sample efficiency. We show that DCG can solve predator-prey tasks that highlight the relative overgeneralization pathology, as well as challenging StarCraft II micromanagement tasks.

Motivation & Objective

• Motivate cooperative MARL and address the exponential growth of the joint action space.
• Propose a deep coordination graph (DCG) that factorizes the joint Q-value into pairwise payoffs plus agent utilities.
• Enable scalable training by sharing parameters across payoff functions and using local information.
• Incorporate low-rank approximations to handle large action spaces efficiently.
• Explore both centralized training with decentralized execution and potential privileged information during training.

Proposed method

• Represent the Q-value as a coordination graph with pairwise payoffs f^ij and per-agent utilities f^i.
• Share parameters across all payoff and utility functions via a common recurrent network, conditioned on agent histories.
• Use a low-rank factorization of payoff matrices f^ij to reduce outputs to 2KA (K is rank, A is action count).
• Apply message passing (max-plus/belief propagation) to maximize the local greedy actions given the graph topology.
• Optionally add a privileged bias v^φ(s) during training (DCG-S) for global state information.
• Extend to permutation-invariant payoffs by averaging f^ij and f^ji to allow topology transfer.

Experimental results

Research questions

• RQ1 Can a factorized Q-function over a coordination graph capture coordinated actions better than fully decentralized baselines?
• RQ2 Does parameter sharing and low-rank payoff approximation improve sample efficiency in large MARL settings?
• RQ3 How do different graph topologies (fully connected, cycle, line, star) affect learning reliability and performance?
• RQ4 Can DCG outperform state-of-the-art methods like QMIX, VDN, QTRAN, and IQL on complex tasks?
• RQ5 Does incorporating privileged training information improve performance (DCG-S)?”],
• RQ6 key_findings）（
• RQ7 table_headers empty?
• RQ8 table_rows empty?

Key findings

• DCG outperforms baseline MARL methods on challenging predator-prey and StarCraft II micromanagement tasks.
• Fully connected DCG reliably solves tasks with strong punishment for uncoordinated actions, while non-shared or topology-limited DCGs show higher variance or fail.
• Low-rank payoff approximations substantially improve sample efficiency without significant loss of performance.
• DCG can represent joint action values better than VDN, QMIX, IQL, and even QTRAN in some settings, particularly under coordinated exploration.
• Graph topology strongly influences reliability; fully connected graphs perform best, while sparse topologies exhibit higher variance across seeds.