[Paper Review] Adaptive Value Decomposition with Greedy Marginal Contribution Computation for Cooperative Multi-Agent Reinforcement Learning

Shanqi Liu, Yujing Hu+6 more|arXiv (Cornell University)|Feb 14, 2023

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes AVGM, a novel explicit credit assignment method for cooperative multi-agent reinforcement learning that uses adaptive value decomposition and greedy marginal contribution computation to solve non-monotonic cooperative tasks. By enabling interpretable, scalable, and high-performing cooperation in complex environments, AVGM achieves significant performance gains over existing value-based methods while maintaining linear time complexity through an action encoder.

ABSTRACT

Real-world cooperation often requires intensive coordination among agents simultaneously. This task has been extensively studied within the framework of cooperative multi-agent reinforcement learning (MARL), and value decomposition methods are among those cutting-edge solutions. However, traditional methods that learn the value function as a monotonic mixing of per-agent utilities cannot solve the tasks with non-monotonic returns. This hinders their application in generic scenarios. Recent methods tackle this problem from the perspective of implicit credit assignment by learning value functions with complete expressiveness or using additional structures to improve cooperation. However, they are either difficult to learn due to large joint action spaces or insufficient to capture the complicated interactions among agents which are essential to solving tasks with non-monotonic returns. To address these problems, we propose a novel explicit credit assignment method to address the non-monotonic problem. Our method, Adaptive Value decomposition with Greedy Marginal contribution (AVGM), is based on an adaptive value decomposition that learns the cooperative value of a group of dynamically changing agents. We first illustrate that the proposed value decomposition can consider the complicated interactions among agents and is feasible to learn in large-scale scenarios. Then, our method uses a greedy marginal contribution computed from the value decomposition as an individual credit to incentivize agents to learn the optimal cooperative policy. We further extend the module with an action encoder to guarantee the linear time complexity for computing the greedy marginal contribution. Experimental results demonstrate that our method achieves significant performance improvements in several non-monotonic domains.

Motivation & Objective

• To address the limitation of value decomposition methods in handling non-monotonic cooperative tasks where traditional monotonic value functions fail.
• To improve credit assignment in cooperative MARL by introducing an explicit, interpretable mechanism based on greedy marginal contribution.
• To ensure scalability in large-agent environments by designing a linear-time complexity computation module via an action encoder.
• To maintain policy interpretability and rationality for real-world deployment, especially in safety-critical applications.
• To demonstrate generalization and robustness across varying numbers of agents and non-monotonic reward structures.

Proposed method

• Proposes an adaptive value decomposition that models individual utility as a distribution over latent actions, dynamically incorporating observable agents' actions.
• Introduces a greedy marginal contribution computation from the decomposed value function to assign explicit credit to each agent based on its contribution to joint outcomes.
• Employs an action encoder to compress joint action representations, ensuring linear time complexity for marginal contribution computation.
• Uses centralized training with decentralized execution (CTDE), where each agent learns its policy based on its own observation and the greedy marginal contribution.
• Applies the Individual Global Maximum (IGM) principle to enable decentralized execution using the decomposed value function.
• Leverages a latent action space to model complex, non-monotonic interactions among agents without requiring full joint action space enumeration.

Figure 1. Left: The illustration of Eq. ( 1 ). The orange area is the payoff matrixes that can be solved by methods using $Q_{i}(\tau_{i},a_{i})$ as policy. Right: Payoff matrix of the bimanual lifting example.

Experimental results

Research questions

• RQ1 Can explicit credit assignment via greedy marginal contribution improve learning in cooperative MARL tasks with non-monotonic returns?
• RQ2 How does AVGM perform compared to implicit credit assignment methods in non-monotonic environments?
• RQ3 Does the action encoder in AVGM maintain linear time complexity while preserving performance in large-scale scenarios?
• RQ4 Can AVGM generalize zero-shot to environments with varying numbers of agents?
• RQ5 To what extent does AVGM improve interpretability and rationality of agent behaviors in cooperative tasks?

Key findings

• AVGM achieves significant performance improvements in non-monotonic environments such as Lift and Pursuit, outperforming baseline methods including QMIX and VDN.
• The method maintains stable and consistent learning across different agent counts, demonstrating robustness and scalability in large-scale settings with up to 12 agents.
• Ablation studies show that removing the greedy marginal contribution or using actual marginal contribution leads to significant performance drops in non-monotonic tasks, confirming its critical role.
• The action encoder enables linear time complexity, making AVGM feasible for large-scale scenarios where full joint action space enumeration is intractable.
• AVGM maintains strong interpretability, as evidenced by consistent perception of environment states and rational agent behavior across training.
• Zero-shot transfer across different agent counts (3–5) shows minimal performance degradation, indicating strong generalization capability.

Figure 2. The architecture of our method. Left (Blue): The architecture for total Q-values. Middle (Green): The framework of the non-monotonic critic $Q_{c}^{i}(\tau_{i},u_{i}^{-},a_{i})$ . Upper Right (Red): The framework of greedy marginal contribution and $Q_{p}^{i}(\tau_{i},a_{i})$ . Lower Right