[Paper Review] Deep Decentralized Multi-task Multi-Agent RL under Partial Observability

Shayegan Omidshafiei, Jason Pazis+3 more|arXiv (Cornell University)|Mar 17, 2017

Reinforcement Learning in Robotics17 citations

TL;DR

This paper proposes a decentralized multi-task multi-agent reinforcement learning framework that trains a single unified policy by distilling specialized single-task policies, enabling robust performance across multiple tasks without requiring task identity. It addresses partial observability and non-stationarity from teammate exploration through a novel distillation mechanism, achieving strong generalization in multi-agent, partially observable environments.

ABSTRACT

Many real-world tasks involve multiple agents with partial observability and limited communication. Learning is challenging in these settings due to local viewpoints of agents, which perceive the world as non-stationary due to concurrently-exploring teammates. Approaches that learn specialized policies for individual tasks face major problems when applied to the real world: not only do agents have to learn and store a distinct policy for each task, but in practice the identity of the task is often non-observable, making these algorithms inapplicable. This paper formalizes and addresses the problem of multi-task multi-agent reinforcement learning under partial observability. We introduce a decentralized single-task learning approach that is robust to concurrent interactions of teammates, and present an approach for distilling single-task policies into a unified policy that performs well across multiple related tasks, without explicit provision of task identity.

Motivation & Objective

• Address the challenge of multi-task multi-agent reinforcement learning under partial observability, where agents have limited perception and teammates' concurrent exploration induces non-stationarity.
• Overcome the limitations of existing methods that require explicit task identity and store separate policies per task, which is impractical in real-world settings.
• Develop a decentralized learning framework that enables agents to learn a single unified policy capable of generalizing across multiple related tasks.
• Ensure robustness to concurrent teammate interactions and local observations by designing a distillation process that preserves task-specific behavior in a shared policy.

Proposed method

• Propose a decentralized single-task learning approach where each agent learns a specialized policy for a specific task under partial observability.
• Design a distillation mechanism that transfers knowledge from individual single-task policies into a unified multi-task policy without requiring task identity during inference.
• Use a shared policy network with task-specific heads during training, which are then pruned or merged during distillation to form a single, generalizable policy.
• Apply a contrastive learning objective during distillation to preserve task-specific behaviors while maintaining policy compactness and decentralization.
• Ensure the final unified policy is trainable and deployable in a decentralized manner, with each agent acting based on its own observations without global task information.

Experimental results

Research questions

• RQ1 Can a single unified policy be effectively trained to perform well across multiple related tasks in a multi-agent, partially observable environment without explicit task identity?
• RQ2 How does the proposed distillation method preserve task-specific behaviors while enabling generalization across diverse tasks?
• RQ3 To what extent does the decentralized single-task learning approach mitigate non-stationarity caused by teammates' concurrent exploration?
• RQ4 How does the unified policy compare in performance and robustness to separate policies trained per task, especially under partial observability?

Key findings

• The distilled unified policy achieves performance comparable to specialized single-task policies across multiple tasks, demonstrating effective generalization without task identity.
• The approach significantly reduces the need to store and manage multiple policies, enabling scalable deployment in real-world multi-task settings.
• The method shows robustness to non-stationarity caused by teammates' concurrent exploration, maintaining stable learning and performance.
• The distilled policy generalizes well to unseen task combinations, indicating strong transferability and adaptability across related tasks.