[論文レビュー] Multi-Agent Deep Reinforcement Learning Under Constrained Communications
DG-MAPPO is a fully distributed MARL framework that uses Distributed Graph Attention Networks to infer global state via multi-hop peer communication, matching or exceeding CTDE baselines without centralized training.
Centralized training with decentralized execution (CTDE) has been the dominant paradigm in multi-agent reinforcement learning (MARL), but its reliance on global state information during training introduces scalability, robustness, and generalization bottlenecks. Moreover, in practical scenarios such as adding/dropping teammates or facing environment dynamics that differ from the training, CTDE methods can be brittle and costly to retrain, whereas distributed approaches allow agents to adapt using only local information and peer-to-peer communication. We present a distributed MARL framework that removes the need for centralized critics or global information. Firstly, we develop a novel Distributed Graph Attention Network (D-GAT) that performs global state inference through multi-hop communication, where agents integrate neighbor features via input-dependent attention weights in a fully distributed manner. Leveraging D-GAT, we develop the distributed graph-attention MAPPO (DG-MAPPO) -- a distributed MARL framework where agents optimize local policies and value functions using local observations, multi-hop communication, and shared/averaged rewards. Empirical evaluation on the StarCraftII Multi-Agent Challenge, Google Research Football, and Multi-Agent Mujoco demonstrates that our method consistently outperforms strong CTDE baselines, achieving superior coordination across a wide range of cooperative tasks with both homogeneous and heterogeneous teams. Our distributed MARL framework provides a principled and scalable solution for robust collaboration, eliminating the need for centralized training or global observability. To the best of our knowledge, DG-MAPPO appears to be the first to fully eliminate reliance on privileged centralized information, enabling agents to learn and act solely through peer-to-peer communication.
研究の動機と目的
- Motivate the need for distributed MARL beyond CTDE due to scalability and robustness constraints of global state reliance.
- Propose a fully distributed framework that learns policies and value functions using only local observations and neighbor communication.
- Introduce D-GAT for multi-hop state inference and DG-MAPPO for distributed policy optimization without centralized critics.
- Demonstrate robustness and scalability of the approach on diverse cooperative benchmarks.
- Show that limited-hop communication can match or surpass CTDE performance in large teams.
提案手法
- Introduce Distributed Graph Attention Networks (D-GAT) where each agent maintains its own attention parameters and performs multi-hop message passing.
- Update D-GAT parameters via decentralized stochastic gradient descent with neighbor averaging to encourage consensus (D-SGD).
- Add a consensus regularization term to align neighboring agents’ representations.
- Build DG-MAPPO by integrating D-GAT inferred global state with local observations into per-agent PPO-style actor-critic updates.
- Use local observations concatenated with D-GAT outputs as input to per-agent actors and critics, trained with GAE-based advantages and clipped PPO objective.
- Evaluate on SMAC, Google Research Football, and Multi-Agent MuJoCo to compare with CTDE baselines.

実験結果
リサーチクエスチョン
- RQ1Can fully distributed MARL achieve competitive coordination without centralized training or global observability?
- RQ2How does multi-hop peer communication (via D-GAT) affect global state inference and policy learning under varying connectivity?
- RQ3What is the impact of hop count on performance and communication overhead in large teams?
- RQ4Does averaging/consensus of D-GAT parameters improve stability and generalization across heterogeneous agent teams?
主な発見
- DG-MAPPO matches or exceeds CTDE baselines (MAPPO, MAT-Dec, HAPPO) on SMAC across diverse tasks.
- DG-MAPPO scales to large teams (up to 25 agents) and remains competitive under sparse communication topologies.
- Performance improves with a small number of hops (K up to N/2 or 1 often suffices) with marginal overhead.
- In GFootball and Multi-Agent MuJoCo, DG-MAPPO achieves competitive returns compared to CTDE baselines, including in continuous-control tasks.
- The approach eliminates reliance on privileged centralized information during training and execution, enabling robust coordination with local observations and peer-to-peer communication.

この論文を読む
Nubintのビューアで原文を開き、読みながら気になる箇所をすぐに質問できます。
Nubintでこの論文を読むこのレビューはAIが論文の原文をもとに作成しました。引用する前に原文をご確認ください。