[Paper Review] Celebrating Diversity in Shared Multi-Agent Reinforcement Learning

Chenghao Li, Tonghan Wang+4 more|arXiv (Cornell University)|Jun 4, 2021

Reinforcement Learning in RoboticsComputer Science46 references49 citations

TL;DR

This paper introduces diversity in both optimization and representation for shared multi-agent reinforcement learning using an information-theoretic intrinsic reward and agent-specific modules with L1 regularization, achieving state-of-the-art performance on Google Research Football and StarCraft II micromanagement tasks.

ABSTRACT

Recently, deep multi-agent reinforcement learning (MARL) has shown the promise to solve complex cooperative tasks. Its success is partly because of parameter sharing among agents. However, such sharing may lead agents to behave similarly and limit their coordination capacity. In this paper, we aim to introduce diversity in both optimization and representation of shared multi-agent reinforcement learning. Specifically, we propose an information-theoretical regularization to maximize the mutual information between agents' identities and their trajectories, encouraging extensive exploration and diverse individualized behaviors. In representation, we incorporate agent-specific modules in the shared neural network architecture, which are regularized by L1-norm to promote learning sharing among agents while keeping necessary diversity. Empirical results show that our method achieves state-of-the-art performance on Google Research Football and super hard StarCraft II micromanagement tasks.

Motivation & Objective

• Motivate and enable diverse yet coordinated behaviors in fully cooperative MARL beyond naive parameter sharing.
• Balance diversity and sharing to improve exploration and coordination in challenging tasks.
• Develop an information-theoretic objective to distinguish agent identities through trajectories to promote individuality.
• Introduce a network structure that combines shared knowledge with non-shared components under L1 regularization.

Proposed method

• Propose an identity-aware diversity objective based on mutual information between agent identity and trajectories to drive exploration and differentiation.
• Use Boltzmann softmax over local Q-values to upper-bound the action-diversity term and optimize a tractable lower bound.
• Introduce a variational posterior q_phi to bound the observation-diversity term and provide intrinsic rewards r^I.
• Decompose each agent’s Q-function into a shared part Q^S and an individual part Q^I, with L1 regularization on Q^I to encourage necessary sharing.
• Employ a QPLEX-based mixing architecture to train with a TD loss that combines environment reward and intrinsic diversity reward.

Experimental results

Research questions

• RQ1 How can diversity be introduced into shared MARL without sacrificing the benefits of parameter sharing?
• RQ2 Can an information-theoretic objective promote individuality among agents while maintaining cooperative performance?
• RQ3 What network architecture and regularization strategies best balance shared knowledge and individual specialization?
• RQ4 Do intrinsic diversity rewards improve performance on challenging MARL benchmarks like GRF and SMAC?

Key findings

• The proposed method achieves state-of-the-art performance on StarCraft II micromanagement super hard maps and challenging Google Research Football tasks such as academy_3_vs_1_with_keeper, academy_counterattack_hard, and 3_vs_1_with_keeper_(full_field).
• Identity-aware diversity plus a partially shared network with an L1-regularized non-shared Q-function yields sophisticated cooperative strategies not seen in baselines.
• Intrinsic rewards based on mutual information between identities and trajectories, action-diversity (via SoftMax over Q-values), and observation-diversity (via a variational bound) drive diverse behaviors.
• A dynamic balance between diversity and sharing emerges: agents learn diverse off-ball movements and coordinated on-ball actions, demonstrating improved coordination under difficult maps.
• Ablation studies show removing components degrades performance, underscoring the necessity of both identity-aware diversity and L1 regularization.