[Paper Review] MAVEN: Multi-Agent Variational Exploration

Anuj Mahajan, Tabish Rashid+2 more|arXiv (Cornell University)|Oct 16, 2019

Reinforcement Learning in RoboticsComputer Science76 citations

TL;DR

MAVEN combines value-based MARL with a latent-variable driven hierarchical policy to enable committed, temporally extended exploration under centralised training with decentralised execution, improving performance on SMAC compared to QMIX and others.

ABSTRACT

Centralised training with decentralised execution is an important setting for cooperative deep multi-agent reinforcement learning due to communication constraints during execution and computational tractability in training. In this paper, we analyse value-based methods that are known to have superior performance in complex environments [43]. We specifically focus on QMIX [40], the current state-of-the-art in this domain. We show that the representational constraints on the joint action-values introduced by QMIX and similar methods lead to provably poor exploration and suboptimality. Furthermore, we propose a novel approach called MAVEN that hybridises value and policy-based methods by introducing a latent space for hierarchical control. The value-based agents condition their behaviour on the shared latent variable controlled by a hierarchical policy. This allows MAVEN to achieve committed, temporally extended exploration, which is key to solving complex multi-agent tasks. Our experimental results show that MAVEN achieves significant performance improvements on the challenging SMAC domain [43].

Motivation & Objective

• Investigate how representational constraints in monotonic value function factorisation affect exploration in CTDE MARL.
• Develop MAVEN to enable diverse, committed exploration while respecting decentralised execution.
• Show that a latent-variable hierarchical policy can induce multiple exploration modes represented by monotonic Q-functions.
• Demonstrate empirical improvements on the SMAC StarCraft II benchmark and in controlled matrix games.

Proposed method

• Introduce a latent space z that conditions the joint action-value function, creating multiple exploration modes.
• Use a hierarchical policy to control z, while value-based agents optimise Q-values conditioned on z.
• Maximise mutual information between trajectories and z to learn diverse exploration modes via a variational objective J_V.
• Combine Q-learning losses with a variational MI reward and a latent space objective to train the model end-to-end.
• At test time, sample z at episode start and perform decentralised argmax on the corresponding Q-function to select actions.

Experimental results

Research questions

• RQ1 Can monotonic value function approximations (e.g., QMIX) suffer from suboptimal exploration due to representational constraints?
• RQ2 Can a latent-variable hierarchical policy enable committed, temporally extended exploration within CTDE MARL?
• RQ3 Does maximizing mutual information between trajectories and latent modes yield diverse, useful exploration behaviors?
• RQ4 Do latent-conditioned, monotonic Q-functions improve performance on challenging MARL benchmarks like SMAC compared to existing methods?

Key findings

• MAVEN achieves significant performance gains on SMAC compared to QMIX and QTRAN on several maps, especially harder ones.
• A latent space z enables multiple exploration modes, each forming a monotonic Q-function conditioned on z, facilitating committed exploration.
• Mutual information between trajectories and z promotes diverse exploration modes and improves learning efficiency.
• Ablations show learned hierarchical policies over z outperform fixed or uniform z distributions, and MI loss contributes to better diversity.