[Paper Review] Multi-Agent Policy Transfer via Task Relationship Modeling

Rongjun Qin, Feng Chen+6 more|arXiv (Cornell University)|Mar 9, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes MATTAR, a multi-agent reinforcement learning framework that models task relationships through effect-based task representations to enable efficient policy transfer across cooperative tasks. By learning a shared, task-relationship-aware representation space via a hypernetwork and alternating fixed training, MATTAR transfers knowledge from a few source tasks to unseen tasks with high success rates, outperforming baselines and enabling effective fine-tuning on new tasks.

ABSTRACT

Team adaptation to new cooperative tasks is a hallmark of human intelligence, which has yet to be fully realized in learning agents. Previous work on multi-agent transfer learning accommodate teams of different sizes, heavily relying on the generalization ability of neural networks for adapting to unseen tasks. We believe that the relationship among tasks provides the key information for policy adaptation. In this paper, we try to discover and exploit common structures among tasks for more efficient transfer, and propose to learn effect-based task representations as a common space of tasks, using an alternatively fixed training scheme. We demonstrate that the task representation can capture the relationship among tasks, and can generalize to unseen tasks. As a result, the proposed method can help transfer learned cooperation knowledge to new tasks after training on a few source tasks. We also find that fine-tuning the transferred policies help solve tasks that are hard to learn from scratch.

Motivation & Objective

• To address the challenge of transferring cooperative policies across multi-agent tasks with varying agent counts and input lengths.
• To improve transfer efficiency by explicitly modeling similarities and differences among tasks through task representations.
• To enable knowledge transfer to unseen tasks using only a few source tasks, without requiring task-specific retraining from scratch.
• To provide a unified framework for multi-agent transfer learning that generalizes better than methods relying solely on neural network inductive bias.

Proposed method

• The method learns effect-based task representations using a forward model that predicts next-state, observation, and reward from current state, action, and observation.
• A shared hypernetwork (representation explainer) generates forward model parameters conditioned on task-specific representations, enabling parameter sharing across tasks.
• An alternative-fixed training scheme fixes task representations as orthogonal vectors during training and updates the representation explainer, then fixes the explainer and fine-tunes the new task representation during inference.
• Policies are conditioned on task representations and designed to be population- and input-length-invariant, enabling adaptation to varying team sizes.
• For unseen tasks, the representation is learned as a linear combination of source task representations by backpropagating through the fixed forward model.
• The framework integrates a population-invariant architecture with self-attention to enhance representational capacity and generalization.

Experimental results

Research questions

• RQ1 Can task relationships be effectively modeled to improve multi-agent policy transfer beyond neural network generalization?
• RQ2 Can a shared representation space capture both similarities and differences among diverse cooperative multi-agent tasks?
• RQ3 Does learning a task representation as a linear combination of source tasks enable effective transfer to unseen tasks?
• RQ4 Can fine-tuning the transferred policy lead to better performance than training from scratch on new tasks?
• RQ5 Does training on multiple source tasks jointly improve transfer performance compared to individual training?

Key findings

• MATTAR achieves superior performance on unseen cooperative tasks, with success rates significantly higher than state-of-the-art baselines like QMIX and QPLEX, especially on challenging maps like MMM2.
• Fine-tuning the transferred policy on unseen tasks leads to better performance than training from scratch, demonstrating that the task representation provides a strong policy initialization.
• The method outperforms single-task learning algorithms even when trained on individual tasks, indicating improved representational capacity.
• The learned linear combination of source task representations shows that the most similar source task typically receives the highest coefficient, confirming that the model captures meaningful task relationships.
• In cases like 10m_vs_12m, multiple source tasks (e.g., 5m and 10m_vs_11m) contribute significantly, suggesting that MATTAR can learn mixed policy knowledge from multiple sources.
• The framework demonstrates comparable performance to baselines on the SMAC benchmark, indicating robustness across diverse cooperative environments.