[Paper Review] More Centralized Training, Still Decentralized Execution: Multi-Agent Conditional Policy Factorization

Jiangxing Wang, Deheng Ye+1 more|arXiv (Cornell University)|Sep 26, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes Multi-Agent Conditional Policy Factorization (MACPF), a method that enables more centralized training by modeling agent dependencies while still supporting decentralized execution through a factorized policy structure. MACPF achieves optimal joint policy performance with faster convergence and superior results over baselines in cooperative MARL environments like SMAC and MPE, by learning dependent local policies that are conditionally factorized into independent policies with correction terms.

ABSTRACT

In cooperative multi-agent reinforcement learning (MARL), combining value decomposition with actor-critic enables agents to learn stochastic policies, which are more suitable for the partially observable environment. Given the goal of learning local policies that enable decentralized execution, agents are commonly assumed to be independent of each other, even in centralized training. However, such an assumption may prohibit agents from learning the optimal joint policy. To address this problem, we explicitly take the dependency among agents into centralized training. Although this leads to the optimal joint policy, it may not be factorized for decentralized execution. Nevertheless, we theoretically show that from such a joint policy, we can always derive another joint policy that achieves the same optimality but can be factorized for decentralized execution. To this end, we propose multi-agent conditional policy factorization (MACPF), which takes more centralized training but still enables decentralized execution. We empirically verify MACPF in various cooperative MARL tasks and demonstrate that MACPF achieves better performance or faster convergence than baselines. Our code is available at https://github.com/PKU-RL/FOP-DMAC-MACPF.

Motivation & Objective

• To address the limitation of existing MARL methods that assume agent independence during training, which restricts joint policy expressiveness and prevents convergence to the global optimum.
• To enable more centralized training by explicitly modeling dependencies among agents during policy learning, improving joint policy quality.
• To ensure decentralized execution is preserved by deriving an independent policy counterpart that matches the performance of the dependent joint policy.
• To empirically validate that introducing agent dependencies during training leads to better performance and faster convergence in complex cooperative environments.

Proposed method

• Proposes multi-agent conditional policy factorization (MACPF), which factorizes the optimal joint policy using a chain rule-based dependency structure during centralized training.
• Introduces a conditional factorization of the joint policy into an independent local policy and a dependency policy correction, enabling decentralized execution.
• Employs soft policy iteration with a mixer network that differs from FOP, allowing the joint Q-value to be factorized based on the chain rule of probabilities.
• Theoretically proves that for any dependent joint policy achieving optimal performance, there exists an equivalent independent joint policy that maintains the same expected return.
• Uses a two-stream architecture: one for learning the dependent policy (for centralized training), and one for the independent policy (for decentralized execution).
• Applies the method in value decomposition frameworks with actor-critic, leveraging the maximum-entropy objective to ensure stochastic policy learning in partially observable settings.

Experimental results

Research questions

• RQ1 Can modeling agent dependencies during centralized training lead to better joint policy optimization in cooperative MARL?
• RQ2 Is it possible to achieve the same optimal performance with a decentralized policy as with a centralized, dependent joint policy?
• RQ3 Does introducing dependencies among agents during training improve convergence speed and final performance in complex cooperative environments?
• RQ4 Can the performance of a dependent joint policy be matched by an independent policy derived through conditional factorization?
• RQ5 How does MACPF compare to state-of-the-art baselines like FOP, QMIX, QPLEX, and MAPPO in terms of win rate and training stability?

Key findings

• MACPF outperforms FOP, QMIX, QPLEX, and MAPPO in 16 out of 18 SMAC maps, with a win rate higher than or equal to the best baseline in 16/18 maps.
• In the challenging 'corridor' map, MACPF achieves a win rate of nearly 70%, while other value decomposition algorithms fail to learn meaningful policies.
• On the MPE benchmark, MACPF achieves a mean reward of -118.24 ± 2.74 in simple spread, significantly outperforming QMIX (-145.93 ± 21.09) and FOP (-125.19 ± 5.42).
• Ablation studies confirm that the performance gain in MACPF is due to modeling dependencies, as MACPF_CONTROL (without dependent policy) underperforms MACPF across all four SMAC maps.
• The independent local policy in MACPF matches the performance of the dependent joint policy throughout training, as shown in Figure 10, validating the theoretical equivalence.
• MACPF converges faster than baselines in hard and super-hard SMAC maps, with a larger performance margin in the more challenging 'myopic' variants (reduced sight range and missing ally info).