[Paper Review] Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning

Jakob Foerster, Francis Song+6 more|arXiv (Cornell University)|Nov 4, 2018

Reinforcement Learning in Robotics29 references49 citations

TL;DR

BAD introduces a public-belief framework (PuB-MDP) enabling scalable counterfactual reasoning in cooperative partially observable multi-agent RL, achieving state-of-the-art Hanabi results.

ABSTRACT

When observing the actions of others, humans make inferences about why they acted as they did, and what this implies about the world; humans also use the fact that their actions will be interpreted in this manner, allowing them to act informatively and thereby communicate efficiently with others. Although learning algorithms have recently achieved superhuman performance in a number of two-player, zero-sum games, scalable multi-agent reinforcement learning algorithms that can discover effective strategies and conventions in complex, partially observable settings have proven elusive. We present the Bayesian action decoder (BAD), a new multi-agent learning method that uses an approximate Bayesian update to obtain a public belief that conditions on the actions taken by all agents in the environment. BAD introduces a new Markov decision process, the public belief MDP, in which the action space consists of all deterministic partial policies, and exploits the fact that an agent acting only on this public belief state can still learn to use its private information if the action space is augmented to be over all partial policies mapping private information into environment actions. The Bayesian update is closely related to the theory of mind reasoning that humans carry out when observing others' actions. We first validate BAD on a proof-of-principle two-step matrix game, where it outperforms policy gradient methods; we then evaluate BAD on the challenging, cooperative partial-information card game Hanabi, where, in the two-player setting, it surpasses all previously published learning and hand-coded approaches, establishing a new state of the art.

Motivation & Objective

• Motivate learning effective communication and conventions in cooperative, partially observable multi-agent settings.
• Introduce a scalable public-belief framework (PuB-MDP) for coordinating agents with private information.
• Develop a method to learn deterministic partial policies conditioned on private observations using deep networks.
• Demonstrate improved performance over baselines in toy and Hanabi experiments.

Proposed method

• Define the public belief B_t as P(f Pri | f pub≤t) and construct the PuB-MDP with state (B_t, f pub) and action space of deterministic partial policies.
• Introduce BAD, a public agent that selects a partial policy based on B_t and f pub, while acting agents use private observations to select environment actions.
• Apply an approximate, factorised Bayesian update to maintain B_t using per-feature likelihoods and sample-based updates.
• Parameterise the BAD policy over partial policies with a factorised structure across private observations, enabling scalable learning with deep networks.
• Share a common random seed to ensure all agents sample the same BAD policy, enabling coordinated exploration as a team.
• Optional self-consistent belief refinement to address feature interactions (V0, V1, V2 beliefs) and improve consistency.

Experimental results

Research questions

• RQ1 Can a public-belief MDP (PuB-MDP) enable scalable learning of communication-based conventions in cooperative MARL with private observations?
• RQ2 Do factorised and approximate Bayesian updates over public beliefs yield practical performance gains in large state spaces like Hanabi?
• RQ3 How does BAD compare to policy-gradient baselines and hand-coded agents in two-player Hanabi?
• RQ4 To what extent do conventions (as opposed to grounded information) contribute to performance in Hanabi under BAD?

Key findings

• BAD outperforms policy-gradient baselines on a proof-of-principle two-step matrix game.
• In two-player Hanabi, BAD achieves 24.174 points on average, surpassing prior learning methods by ~9 points and approaching open-hand cheating performance.
• BAD reaches a high rate of perfect games (58.6% in Table 1) under evaluation.
• Beliefs learned via Bayesian updates reduce uncertainty about hands by about 40% compared to grounded-inference baselines.
• About 40% of information in Hanabi play is conveyed via conventions rather than grounded knowledge, as evidenced by in-game behavior analyses.
• BAD establishes a new state-of-the-art on the Hanabi Learning Environment for two-player self-play.