[Paper Review] LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning

Mingyu Yang, Jian Zhao+4 more|arXiv (Cornell University)|May 5, 2022

Sports Analytics and Performance22 citations

TL;DR

LDSA learns dynamic subtask assignment in cooperative MARL by encoding subtasks, assigning agents by learned abilities, and using a decoder-guided, representation-dependent policy for each subtask, improving performance on SMAC and GRF.

ABSTRACT

Cooperative multi-agent reinforcement learning (MARL) has made prominent progress in recent years. For training efficiency and scalability, most of the MARL algorithms make all agents share the same policy or value network. However, in many complex multi-agent tasks, different agents are expected to possess specific abilities to handle different subtasks. In those scenarios, sharing parameters indiscriminately may lead to similar behavior across all agents, which will limit the exploration efficiency and degrade the final performance. To balance the training complexity and the diversity of agent behavior, we propose a novel framework to learn dynamic subtask assignment (LDSA) in cooperative MARL. Specifically, we first introduce a subtask encoder to construct a vector representation for each subtask according to its identity. To reasonably assign agents to different subtasks, we propose an ability-based subtask selection strategy, which can dynamically group agents with similar abilities into the same subtask. In this way, agents dealing with the same subtask share their learning of specific abilities and different subtasks correspond to different specific abilities. We further introduce two regularizers to increase the representation difference between subtasks and stabilize the training by discouraging agents from frequently changing subtasks, respectively. Empirical results show that LDSA learns reasonable and effective subtask assignment for better collaboration and significantly improves the learning performance on the challenging StarCraft II micromanagement benchmark and Google Research Football.

Motivation & Objective

• Motivate the need to balance training complexity and behavioral diversity in cooperative MARL by decomposing tasks into subtasks.
• Develop a framework to learn subtasks without heavy domain knowledge.
• Enable dynamic, ability-based assignment of agents to subtasks to improve collaboration.
• Learn per-subtask policies coupled with subtask representations to promote sharing within subtasks and diversity across subtasks.

Proposed method

• Introduce a subtask encoder to generate a vector representation for each subtask based on its identity.
• Use an agent trajectory/action-observation history encoder to obtain ability representations and compute cosine similarities to subtask representations for subtask selection via a softmax distribution.
• Apply Straight-Through Gumbel-Softmax to sample subtask assignments in a differentiable manner.
• Use a subtask decoder to map subtask representations to policy parameters, linking subtask identity to distinct policies.
• Incorporate two regularizers: (i) maximize pairwise subtask representation diversity to encourage meaningful decomposition, and (ii) minimize KL divergence of subtask selection distributions across adjacent timesteps to stabilize training.
• Train with a QMIX-style mixing network to obtain a global Q-value while allowing decentralized execution.

Experimental results

Research questions

• RQ1 Can LDSA outperform baselines on challenging cooperative MARL benchmarks?
• RQ2 Do the proposed regularizers improve decomposition quality and training stability?
• RQ3 Which components of LDSA contribute most to performance gains?
• RQ4 Can LDSA provide interpretable and reasonable dynamic subtask assignments over training?

Key findings

• LDSA significantly improves learning performance across 14 SMAC scenarios, surpassing baselines and achieving about a 7% gain in median test win rate on average.
• Regularizers that enforce subtask separation and stabilize subtask switching contribute to performance; removing them degrades results, with the representation-diversity regularizer having a larger impact.
• Ablations show that the ability-based subtask selection contributes most to performance, more than the subtask decoder or simply increasing parameter count.
• Visualizations demonstrate dynamic, rational subtask assignments where agents with similar abilities cluster to solve specific subtasks and share learning effectively.