[Paper Review] UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers

Siyi Hu, Fengda Zhu+2 more|arXiv (Cornell University)|Jan 20, 2021

Reinforcement Learning in RoboticsComputer Science35 references32 citations

TL;DR

UPDeT introduces a universal, transformer-based policy decoupling framework for multi-agent RL that handles variable input/output dimensions and enables fast transfer across tasks, outperforming RNN-based methods.

ABSTRACT

Recent advances in multi-agent reinforcement learning have been largely limited in training one model from scratch for every new task. The limitation is due to the restricted model architecture related to fixed input and output dimensions. This hinders the experience accumulation and transfer of the learned agent over tasks with diverse levels of difficulty (e.g. 3 vs 3 or 5 vs 6 multi-agent games). In this paper, we make the first attempt to explore a universal multi-agent reinforcement learning pipeline, designing one single architecture to fit tasks with the requirement of different observation and action configurations. Unlike previous RNN-based models, we utilize a transformer-based model to generate a flexible policy by decoupling the policy distribution from the intertwined input observation with an importance weight measured by the merits of the self-attention mechanism. Compared to a standard transformer block, the proposed model, named as Universal Policy Decoupling Transformer (UPDeT), further relaxes the action restriction and makes the multi-agent task's decision process more explainable. UPDeT is general enough to be plugged into any multi-agent reinforcement learning pipeline and equip them with strong generalization abilities that enables the handling of multiple tasks at a time. Extensive experiments on large-scale SMAC multi-agent competitive games demonstrate that the proposed UPDeT-based multi-agent reinforcement learning achieves significant results relative to state-of-the-art approaches, demonstrating advantageous transfer capability in terms of both performance and training speed (10 times faster).

Motivation & Objective

• Motivate the need for a universal MARL architecture that works across varying observation and action configurations.
• Propose a transformer-based individual function that decouples policy from input observations.
• Introduce policy decoupling to map matched observation-entity to action-groups without adding new task-specific parameters.
• Enable multi-task transfer and faster adaptation across diverse MARL tasks while preserving explainability.

Proposed method

• Represent observations as observation-entities and embed them with a transformer-based function to compute per-agent Q-values.
• Compute a global Q-function from individual Q-values via a credit assignment function.
• Use self-attention to learn relationships among matched observation-entity and other observations, enabling policy decoupling.
• Partition actions into action-groups matched to observation-entities to allow flexible policy dimensions without extra parameters.
• Incorporate temporal units (global or individual) to handle action-observation history within a Dec-POMDP setting.
• Optimize with standard TD error as in DQN, replacing GRU/LSTM with transformer-based temporal units.

Experimental results

Research questions

• RQ1 Can a single architecture support MARL tasks with varying input/output dimensions without adding new task-specific parameters?
• RQ2 Does policy decoupling via a transformer improve representation learning and transfer across multiple MARL tasks?
• RQ3 How does UPDeT perform when plugged into existing MARL pipelines (VDN, QMIX, QTRAN) in terms of performance and transfer speed?
• RQ4 What is the impact of different temporal unit designs on learning under partial observability?
• RQ5 Is the attention mechanism capable of providing explainable insights into policy decisions in multi-agent settings?

Key findings

• UPDeT substantially outperforms RNN-based models on challenging SMAC scenarios when paired with VDN, QMIX, or QTRAN.
• The approach enables strong transfer capability across tasks and significantly reduces training cost, with reported transfer convergence costs at least 10x faster than GRU-based models and 100x faster than training from scratch.
• Attention-guided policy decoupling yields interpretable strategies, as evidenced by attention maps correlating with strategic stages like Startup, Attack, and Survival in STARCRAFT-like settings.
• UPDeT can be plugged into existing MARL methods with almost no architectural changes and yields notable performance gains across easy to hard scenarios.
• The method scales to large-scale MAS settings and demonstrates robust generalization and transfer in various task sizes.