[Paper Review] RODE: Learning Roles to Decompose Multi-Agent Tasks

Tonghan Wang, Tarun Gupta+4 more|arXiv (Cornell University)|Oct 4, 2020

Reinforcement Learning in Robotics98 references60 citations

TL;DR

RODE automatically discovers roles by clustering actions based on their effects, creating a bi-level learning framework that decomposes multi-agent tasks into smaller, transferable sub-tasks.

ABSTRACT

Role-based learning holds the promise of achieving scalable multi-agent learning by decomposing complex tasks using roles. However, it is largely unclear how to efficiently discover such a set of roles. To solve this problem, we propose to first decompose joint action spaces into restricted role action spaces by clustering actions according to their effects on the environment and other agents. Learning a role selector based on action effects makes role discovery much easier because it forms a bi-level learning hierarchy -- the role selector searches in a smaller role space and at a lower temporal resolution, while role policies learn in significantly reduced primitive action-observation spaces. We further integrate information about action effects into the role policies to boost learning efficiency and policy generalization. By virtue of these advances, our method (1) outperforms the current state-of-the-art MARL algorithms on 10 of the 14 scenarios that comprise the challenging StarCraft II micromanagement benchmark and (2) achieves rapid transfer to new environments with three times the number of agents. Demonstrative videos are available at https://sites.google.com/view/rode-marl .

Motivation & Objective

• Motivate scalable multi-agent learning through role-based decomposition.
• Automatically discover an effective set of roles without handcrafted designs.
• Reduce learning complexity by factoring joint action spaces via action effects.
• Enable fast transfer of learned policies to environments with different numbers of agents/actions.

Proposed method

• Learn action representations that encode action effects on observations and rewards using a forward predictive model.
• Cluster actions in the representation space to form restricted role action spaces.
• Introduce a bi-level hierarchy: a role selector at a high level and role policies operating in restricted action spaces.
• Compute role representations as the average of action representations to inform role selection.
• Learn role policies and a role selector with QMIX-style mixing networks to optimize joint return.
• Train end-to-end with TD losses for both the role selector and role policies, leveraging global rewards.

Experimental results

Research questions

• RQ1 Can action-effect based representations effectively cluster actions for role-based decomposition?
• RQ2 Does restricting role action spaces improve learning efficiency and policy performance in large multi-agent environments?
• RQ3 Can RODE transfer learned policies to tasks with different numbers of agents or actions?
• RQ4 What is the contribution of each component (action representations, restricted action spaces, hierarchical learning) to overall performance?
• RQ5 How well does RODE perform on challenging multi-agent benchmarks like StarCraft II micromanagement?

Key findings

• RODE achieves state-of-the-art performance on 10 of 14 StarCraft II micromanagement maps, including all 9 hard and super hard maps.
• RODE shows rapid transfer to environments with three times more agents than in the training setup.
• Action representations effectively reveal action clusters that correspond to functional similarities (e.g., moving toward/away from enemies, attacking similar unit types).
• Ablations show restricting role action spaces and using action-effect information are crucial for gains over baselines, while full action spaces or random restrictions don’t yield similar benefits.
• RODE’s hierarchical design with a role selector and role policies provides a scalable learning framework when combined with effect-based action factoring.