[Paper Review] Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning

Hengyuan Hu, Jakob Foerster|arXiv (Cornell University)|Apr 30, 2020

Reinforcement Learning in RoboticsComputer Science30 references17 citations

TL;DR

This paper introduces the Simplified Action Decoder (SAD), a deep multi-agent reinforcement learning method that enhances cooperative communication in Hanabi by enabling agents to infer teammates' greedy actions during training. By leveraging centralized training to decode intentions and using an auxiliary state prediction task, SAD achieves state-of-the-art performance in 2–5 player self-play Hanabi settings, resolving the exploration-informativeness trade-off in cooperative multi-agent RL.

ABSTRACT

In recent years we have seen fast progress on a number of benchmark problems in AI, with modern methods achieving near or super human performance in Go, Poker and Dota. One common aspect of all of these challenges is that they are by design adversarial or, technically speaking, zero-sum. In contrast to these settings, success in the real world commonly requires humans to collaborate and communicate with others, in settings that are, at least partially, cooperative. In the last year, the card game Hanabi has been established as a new benchmark environment for AI to fill this gap. In particular, Hanabi is interesting to humans since it is entirely focused on theory of mind, i.e. the ability to effectively reason over the intentions, beliefs and point of view of other agents when observing their actions. Learning to be informative when observed by others is an interesting challenge for Reinforcement Learning (RL): Fundamentally, RL requires agents to explore in order to discover good policies. However, when done naively, this randomness will inherently make their actions less informative to others during training. We present a new deep multi-agent RL method, the Simplified Action Decoder (SAD), which resolves this contradiction exploiting the centralized training phase. During training SAD allows other agents to not only observe the (exploratory) action chosen, but agents instead also observe the greedy action of their team mates. By combining this simple intuition with an auxiliary task for state prediction and best practices for multi-agent learning, SAD establishes a new state of the art for 2-5 players on the self-play part of the Hanabi challenge.

Motivation & Objective

• Address the challenge of balancing exploration and informativeness in cooperative multi-agent reinforcement learning.
• Enable agents to infer teammates' intended actions despite exploratory behavior during training.
• Improve communication efficiency in partially observable, cooperative environments like Hanabi.
• Overcome the inherent contradiction where exploratory actions reduce information sharing during training.
• Establish a new state of the art in self-play Hanabi for 2–5 player settings using a simple yet effective architecture.

Proposed method

• Introduce a centralized training mechanism where agents observe not only their own exploratory actions but also the greedy actions of teammates.
• Use a simplified action decoder head to reconstruct teammates' intended actions from policy outputs.
• Incorporate an auxiliary task for state prediction to improve policy generalization and communication.
• Leverage multi-agent RL best practices, including curriculum learning and value function regularization.
• Train the policy end-to-end with a combination of intrinsic dense rewards and the auxiliary state prediction loss.
• Decouple exploration from communication by allowing agents to infer intentions even when actions are stochastic.

Experimental results

Research questions

• RQ1 Can we improve cooperative multi-agent communication in partially observable environments like Hanabi by decoding teammates' intentions during training?
• RQ2 How does incorporating a centralized decoder for greedy actions affect performance in cooperative multi-agent RL?
• RQ3 To what extent does an auxiliary state prediction task enhance communication and policy learning in cooperative settings?
• RQ4 Can a simple architectural modification resolve the exploration-informativeness trade-off in cooperative multi-agent RL?
• RQ5 Does the proposed method achieve state-of-the-art performance in self-play Hanabi across 2–5 player settings?

Key findings

• SAD achieves a new state of the art in the self-play setting of the Hanabi challenge for 2–5 player games.
• The method effectively decouples exploration from communication by allowing agents to infer teammates' intended actions.
• The auxiliary state prediction task contributes to improved policy generalization and communication efficiency.
• The approach resolves the fundamental conflict between exploratory behavior and informative action selection during training.
• SAD demonstrates strong performance gains over prior methods without requiring complex architectural modifications.
• The centralized training phase enables effective intention decoding, significantly improving team-level cooperation.