[论文解读] Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning

Hengyuan Hu, Jakob Foerster|arXiv (Cornell University)|Apr 30, 2020

Reinforcement Learning in Robotics参考文献 30被引用 17

一句话总结

本文提出了简化动作解码器（SAD），一种深度多智能体强化学习方法，通过在训练期间使智能体能够推断队友的贪婪动作，从而提升汉拉比游戏中合作通信的效果。通过利用集中式训练来解码意图，并引入辅助状态预测任务，SAD在2–5名玩家的自对弈汉拉比游戏设置中实现了最先进性能，解决了合作多智能体强化学习中的探索-信息量权衡问题。

ABSTRACT

In recent years we have seen fast progress on a number of benchmark problems in AI, with modern methods achieving near or super human performance in Go, Poker and Dota. One common aspect of all of these challenges is that they are by design adversarial or, technically speaking, zero-sum. In contrast to these settings, success in the real world commonly requires humans to collaborate and communicate with others, in settings that are, at least partially, cooperative. In the last year, the card game Hanabi has been established as a new benchmark environment for AI to fill this gap. In particular, Hanabi is interesting to humans since it is entirely focused on theory of mind, i.e. the ability to effectively reason over the intentions, beliefs and point of view of other agents when observing their actions. Learning to be informative when observed by others is an interesting challenge for Reinforcement Learning (RL): Fundamentally, RL requires agents to explore in order to discover good policies. However, when done naively, this randomness will inherently make their actions less informative to others during training. We present a new deep multi-agent RL method, the Simplified Action Decoder (SAD), which resolves this contradiction exploiting the centralized training phase. During training SAD allows other agents to not only observe the (exploratory) action chosen, but agents instead also observe the greedy action of their team mates. By combining this simple intuition with an auxiliary task for state prediction and best practices for multi-agent learning, SAD establishes a new state of the art for 2-5 players on the self-play part of the Hanabi challenge.

研究动机与目标

• 解决合作多智能体强化学习中探索与信息量之间的平衡挑战。
• 使智能体能够在训练期间即使面对探索性行为，也能推断出队友的意图动作。
• 提升在部分可观察合作环境（如汉拉比）中的通信效率。
• 克服探索性动作在训练期间降低信息共享的固有矛盾。
• 通过一种简单但有效的架构，在2–5名玩家的自对弈汉拉比游戏中建立新的最先进水平。

提出的方法

• 引入集中式训练机制，使智能体不仅能观察自身探索性动作，还能观察队友的贪婪动作。
• 使用简化动作解码头，从策略输出中重建队友的意图动作。
• 引入辅助状态预测任务，以提升策略泛化能力和通信效果。
• 采用多智能体强化学习的最佳实践，包括课程学习和价值函数正则化。
• 通过内在密集奖励与辅助状态预测损失相结合的方式，端到端训练策略。
• 通过允许智能体在动作具有随机性时仍能推断意图，实现探索与通信的解耦。

实验结果

研究问题

• RQ1 我们能否通过在训练期间解码队友意图，来改善汉拉比等部分可观察环境中的合作多智能体通信？
• RQ2 在合作多智能体强化学习中，引入用于贪婪动作的集中式解码器在性能上会产生何种影响？
• RQ3 辅助状态预测任务在多大程度上能增强合作设置中的通信与策略学习？
• RQ4 简单的架构修改是否能够解决合作多智能体强化学习中探索-信息量权衡问题？
• RQ5 所提出的方法是否在2–5名玩家的自对弈汉拉比游戏中实现了最先进性能？

主要发现

• SAD在2–5名玩家的自对弈汉拉比挑战设置中实现了新的最先进水平。
• 该方法通过使智能体能够推断队友的意图动作，有效实现了探索与通信的解耦。
• 辅助状态预测任务显著提升了策略泛化能力和通信效率。
• 该方法解决了训练期间探索性行为与信息性动作选择之间的根本性矛盾。
• SAD在无需复杂架构修改的前提下，相对于先前方法表现出显著的性能提升。
• 集中式训练阶段使意图解码更加有效，显著提升了团队协作水平。