[论文解读] Learning Implicit Credit Assignment for Multi-Agent Actor-Critic.

Meng Zhou, Ziyu Liu+3 more|arXiv (Cornell University)|Jul 6, 2020

Reinforcement Learning in Robotics参考文献 14被引用 14

一句话总结

该论文提出LICA，一种多智能体强化学习算法，通过基于超网络的集中式评论家隐式解决信用分配问题，同时利用自适应熵正则化保持探索能力。LICA在多智能体粒子环境和星际争霸II微操控制任务中均优于先前方法。

ABSTRACT

We present a new policy-based multi-agent reinforcement learning algorithm that implicitly addresses the credit assignment problem under fully cooperative settings. Our key motivation is that credit assignment may not require an explicit formulation as long as (1) the policy gradients of a trained, centralized critic carry sufficient information for the decentralized agents to maximize the critic estimate through optimal cooperation and (2) a sustained level of agent exploration is enforced throughout training. In this work, we achieve the former by formulating the centralized critic as a hypernetwork such that the latent state representation is now fused into the policy gradients through its multiplicative association with the agent policies, and we show that this is key to learning optimal joint actions that may otherwise require explicit credit assignment. To achieve the latter, we further propose a practical technique called adaptive entropy regularization where magnitudes of the policy gradients from the entropy term are dynamically rescaled to sustain consistent levels of exploration throughout training. Our final algorithm, which we call LICA, is evaluated on several benchmarks including the multi-agent particle environments and a set of challenging StarCraft II micromanagement tasks, and we show that LICA significantly outperforms previous methods.

研究动机与目标

• 解决合作性多智能体强化学习中的信用分配问题，而无需显式奖励分解。
• 通过来自集中式评论家的策略梯度，使去中心化智能体能够学习最优联合动作。
• 通过动态熵正则化技术，在训练过程中保持一致的探索水平。
• 在如星际争霸II微操控制任务等复杂多智能体基准上提升性能。

提出的方法

• 将集中式评论家建模为超网络，通过关注潜在状态表示来生成策略梯度。
• 将潜在状态表示以乘法方式融合进策略梯度中，通过共享价值估计实现隐式信用分配。
• 引入自适应熵正则化，动态调整熵项的大小，以维持探索能力。
• 使用由评论家价值估计和熵正则化共同指导的策略梯度来训练策略网络。
• 利用集中式评论家为所有智能体提供价值目标，实现在无需显式信用分配情况下的联合动作优化。

实验结果

研究问题

• RQ1 能否通过基于超网络的评论家，将潜在状态信息嵌入策略梯度中，实现隐式信用分配？
• RQ2 通过自适应熵正则化维持一致的探索水平，是否能提升多智能体设置下的学习稳定性和性能？
• RQ3 在无需显式信用分配的情况下，基于策略的多智能体算法能否在复杂合作任务中超越现有方法？
• RQ4 所提出方法在如星际争霸II微操控制任务等复杂环境中的可扩展性如何？

主要发现

• LICA在多智能体粒子环境基准测试中显著优于先前方法。
• 该算法在具有挑战性的星际争霸II微操控制任务中表现出色，证明了其在复杂合作场景中的鲁棒性。
• 自适应熵正则化成功在训练全程维持了稳定的探索水平，提升了策略学习的稳定性。
• 基于超网络的评论家实现了隐式信用分配，使智能体能够在无需显式奖励分解的情况下学习最优联合动作。