[论文解读] Adaptive Value Decomposition with Greedy Marginal Contribution Computation for Cooperative Multi-Agent Reinforcement Learning

Shanqi Liu, Yujing Hu+6 more|arXiv (Cornell University)|Feb 14, 2023

Reinforcement Learning in Robotics被引用 4

一句话总结

本文提出 AVGM，一种用于合作多智能体强化学习的新型显式信用分配方法，通过自适应价值分解与贪婪边际贡献计算，解决非单调合作任务。AVGM 在复杂环境中实现了可解释性、可扩展性与高性能的合作，通过动作编码器保持线性时间复杂度，显著优于现有基于价值的方法。

ABSTRACT

Real-world cooperation often requires intensive coordination among agents simultaneously. This task has been extensively studied within the framework of cooperative multi-agent reinforcement learning (MARL), and value decomposition methods are among those cutting-edge solutions. However, traditional methods that learn the value function as a monotonic mixing of per-agent utilities cannot solve the tasks with non-monotonic returns. This hinders their application in generic scenarios. Recent methods tackle this problem from the perspective of implicit credit assignment by learning value functions with complete expressiveness or using additional structures to improve cooperation. However, they are either difficult to learn due to large joint action spaces or insufficient to capture the complicated interactions among agents which are essential to solving tasks with non-monotonic returns. To address these problems, we propose a novel explicit credit assignment method to address the non-monotonic problem. Our method, Adaptive Value decomposition with Greedy Marginal contribution (AVGM), is based on an adaptive value decomposition that learns the cooperative value of a group of dynamically changing agents. We first illustrate that the proposed value decomposition can consider the complicated interactions among agents and is feasible to learn in large-scale scenarios. Then, our method uses a greedy marginal contribution computed from the value decomposition as an individual credit to incentivize agents to learn the optimal cooperative policy. We further extend the module with an action encoder to guarantee the linear time complexity for computing the greedy marginal contribution. Experimental results demonstrate that our method achieves significant performance improvements in several non-monotonic domains.

研究动机与目标

• 解决价值分解方法在处理传统单调价值函数失效的非单调合作任务时的局限性。
• 通过引入基于贪婪边际贡献的显式、可解释机制，改进合作多智能体强化学习中的信用分配。
• 通过设计基于动作编码器的线性时间复杂度计算模块，确保在大规模智能体环境中的可扩展性。
• 通过保持策略的可解释性与合理性，确保在安全关键应用等实际部署场景中的适用性。
• 在不同智能体数量与非单调奖励结构下，展示泛化性与鲁棒性。

提出的方法

• 提出一种自适应价值分解方法，将个体效用建模为潜在动作的分布，动态整合可观测智能体的动作。
• 从分解后的价值函数中引入贪婪边际贡献计算，基于智能体对联合结果的贡献，显式分配信用。
• 采用动作编码器压缩联合动作表示，确保边际贡献计算的线性时间复杂度。
• 采用集中式训练、分布式执行（CTDE）范式，每个智能体基于自身观测与贪婪边际贡献学习策略。
• 应用个体全局最大（IGM）原则，利用分解后的价值函数实现去中心化执行。
• 利用潜在动作空间建模智能体之间复杂且非单调的交互关系，而无需枚举完整的联合动作空间。

Figure 1. Left: The illustration of Eq. ( 1 ). The orange area is the payoff matrixes that can be solved by methods using $Q_{i}(\tau_{i},a_{i})$ as policy. Right: Payoff matrix of the bimanual lifting example.

实验结果

研究问题

• RQ1 通过贪婪边际贡献实现的显式信用分配，是否能提升非单调回报合作多智能体强化学习任务的学习性能？
• RQ2 AVGM 在非单调环境中与隐式信用分配方法相比表现如何？
• RQ3 AVGM 中的动作编码器是否在保持性能的同时维持线性时间复杂度，适用于大规模场景？
• RQ4 AVGM 是否能实现零样本迁移至不同智能体数量的环境？
• RQ5 AVGM 在多大程度上提升了合作任务中智能体行为的可解释性与合理性？

主要发现

• AVGM 在非单调环境（如 Lift 与 Pursuit）中实现显著性能提升，优于包括 QMIX 与 VDN 在内的基线方法。
• 该方法在不同智能体数量下均保持稳定且一致的学习过程，展现出在最多 12 个智能体的大规模设置中的鲁棒性与可扩展性。
• 消融实验表明，若移除贪婪边际贡献或改用实际边际贡献，非单调任务中的性能显著下降，证实其关键作用。
• 动作编码器实现线性时间复杂度，使 AVGM 在全联合动作空间枚举不可行的大规模场景中具备可行性。
• AVGM 保持了强大的可解释性，表现为训练过程中对环境状态的一致感知与理性智能体行为。
• 在不同智能体数量（3–5）间实现零样本迁移时性能下降极小，表明其具备强大的泛化能力。

Figure 2. The architecture of our method. Left (Blue): The architecture for total Q-values. Middle (Green): The framework of the non-monotonic critic $Q_{c}^{i}(\tau_{i},u_{i}^{-},a_{i})$ . Upper Right (Red): The framework of greedy marginal contribution and $Q_{p}^{i}(\tau_{i},a_{i})$ . Lower Right