[Paper Review] ROMA: Multi-Agent Reinforcement Learning with Emergent Roles

Tonghan Wang, Heng Dong+2 more|arXiv (Cornell University)|Mar 18, 2020

Reinforcement Learning in RoboticsComputer Science59 references32 citations

TL;DR

ROMA introduces emergent, role-conditioned policies in MARL, using identifiability and specialization regularizers to enable dynamic division of labor and improved performance on StarCraft II micromanagement benchmarks.

ABSTRACT

The role concept provides a useful tool to design and understand complex multi-agent systems, which allows agents with a similar role to share similar behaviors. However, existing role-based methods use prior domain knowledge and predefine role structures and behaviors. In contrast, multi-agent reinforcement learning (MARL) provides flexibility and adaptability, but less efficiency in complex tasks. In this paper, we synergize these two paradigms and propose a role-oriented MARL framework (ROMA). In this framework, roles are emergent, and agents with similar roles tend to share their learning and to be specialized on certain sub-tasks. To this end, we construct a stochastic role embedding space by introducing two novel regularizers and conditioning individual policies on roles. Experiments show that our method can learn specialized, dynamic, and identifiable roles, which help our method push forward the state of the art on the StarCraft II micromanagement benchmark. Demonstrative videos are available at https://sites.google.com/view/romarl/.

Motivation & Objective

• Motivate integrating role concepts with MARL to improve learning efficiency in complex tasks.
• Enable emergent roles that are dynamic, identifiable, and specialized across agents.
• Develop a mechanism for role-conditioned policies with tractable optimization.

Proposed method

• Introduce a stochastic role embedding space where each agent's role is drawn from a learnable Gaussian conditioned on local observations.
• Use a role encoder to produce role parameters and a role decoder (hyper-network) to generate policy parameters.
• Impose two regularizers: an identifiable role objective based on variational mutual information, and a specialized role objective using a dissimilarity model with a variational bound.
• Adopt centralized training with decentralized execution using a QMIX-style mixing network to compute a global value.
• Optimize the overall objective with TD loss plus regularizers weighted by lambda_I and lambda_D.
• Demonstrate role emergence and specialization through visualization and ablation analyses on StarCraft II maps.

Experimental results

Research questions

• RQ1 Can learned roles adapt dynamically to changing environments?
• RQ2 Do learned roles lead to sub-task specialization where similar roles share similar policies?
• RQ3 Does sub-task specialization improve MARL performance on challenging benchmarks?
• RQ4 How do roles emerge and evolve during training, and how do they relate to team performance?
• RQ5 Can the dissimilarity model learn meaningful trajectory-based role distinctions?

Key findings

• ROMA learns dynamic role representations that cluster by sub-task and adapt with enemy configurations and agent health.
• Specialization regularizer improves performance and leads to more efficient division of labor than baselines.
• ROMA outperforms baselines on multiple StarCraft II SMAC maps, including heterogeneous and homogeneous agent settings.
• Role embeddings reveal interpretable sub-tasks, such as formation shaping and protective maneuvers, emerging over training.
• The dissimilarity model d_phi captures trajectory-based differences across agents, aligning high dissimilarity with distinct unit types.