[Paper Review] QGNN: Value Function Factorisation with Graph Neural Networks

Ryan Kortvelesy, Amanda Prorok|arXiv (Cornell University)|May 25, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

QGNN proposes a graph neural network-based value factorisation method for multi-agent reinforcement learning that leverages multi-hop message passing to improve credit assignment. By modeling agents as a graph and using a permutation-invariant mixer, QGNN achieves state-of-the-art performance on StarCraft, Estimate Game, and Coalition Structure Generation, outperforming baselines like QMIX-Att and GraphMIX with fewer parameters.

ABSTRACT

In multi-agent reinforcement learning, the use of a global objective is a powerful tool for incentivising cooperation. Unfortunately, it is not sample-efficient to train individual agents with a global reward, because it does not necessarily correlate with an agent's individual actions. This problem can be solved by factorising the global value function into local value functions. Early work in this domain performed factorisation by conditioning local value functions purely on local information. Recently, it has been shown that providing both local information and an encoding of the global state can promote cooperative behaviour. In this paper we propose QGNN, the first value factorisation method to use a graph neural network (GNN) based model. The multi-layer message passing architecture of QGNN provides more representational complexity than models in prior work, allowing it to produce a more effective factorisation. QGNN also introduces a permutation invariant mixer which is able to match the performance of other methods, even with significantly fewer parameters. We evaluate our method against several baselines, including QMIX-Att, GraphMIX, QMIX, VDN, and hybrid architectures. Our experiments include Starcraft, the standard benchmark for credit assignment; Estimate Game, a custom environment that explicitly models inter-agent dependencies; and Coalition Structure Generation, a foundational problem with real-world applications. The results show that QGNN outperforms state-of-the-art value factorisation baselines consistently.

Motivation & Objective

• To address the credit assignment problem in cooperative multi-agent reinforcement learning where global rewards do not correlate with individual actions.
• To improve value function factorisation by incorporating global state information and agent interdependencies through a graph-structured model.
• To reduce parameter count while maintaining or improving performance through a permutation-invariant mixer.
• To demonstrate the importance of representational capacity in communication-based models, especially within the GNN architecture.
• To provide a new metric for evaluating implicit credit assignment by comparing learned local values to known ground-truth values.

Proposed method

• QGNN models agents as nodes in a graph, with edges representing communication links, enabling multi-hop message passing for richer representation learning.
• It employs a graph neural network (GNN) with multi-layer message passing to learn local value functions conditioned on both local observations and global context.
• A permutation-invariant mixer combines the GNN-processed representations to produce individual Q-values, ensuring invariance to agent ordering.
• The architecture supports incremental communication by adjusting graph connectivity, modeling real-world constraints like communication range.
• The method implicitly learns Shapley value-like credit assignment through GNN message passing, validated via a custom environment with known true local values.
• An ablation study evaluates the impact of graph connectivity (communication range) and component contributions, isolating representational complexity as key.

Experimental results

Research questions

• RQ1 Can a GNN-based architecture with multi-hop message passing improve value function factorisation compared to prior methods?
• RQ2 Does a permutation-invariant mixer enable high performance with fewer parameters in cooperative MARL?
• RQ3 How does representational complexity in the GNN component compare to that in the mixer for credit assignment performance?
• RQ4 To what extent can a GNN implicitly learn true local values in a credit assignment setting, as measured by a new evaluation metric?
• RQ5 How does graph connectivity (e.g., communication range) affect the performance of the value factorisation method?

Key findings

• QGNN outperforms state-of-the-art baselines, including QMIX-Att, GraphMIX, QMIX, VDN, and hybrid models, across all evaluated environments.
• In the StarCraft Multi-Agent Challenge, QGNN achieves higher mean returns than all baselines, demonstrating superior credit assignment in complex cooperative tasks.
• On the custom Estimate Game environment, QGNN successfully learns inter-agent dependencies and matches or exceeds performance of methods relying on explicit attention or complex architectures.
• In Coalition Structure Generation, QGNN achieves optimal or near-optimal solutions with significantly fewer parameters than comparable models.
• The ablation study confirms that representational complexity in the GNN is more critical than in the mixer, with GNN depth and message-passing hops having a greater impact on performance.
• The new metric based on known ground-truth local values shows that QGNN's implicit credit assignment closely approximates Shapley values, validating its theoretical alignment with cooperative game theory.