[Paper Review] Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability

Shayegan Omidshafiei, Jason Pazis+3 more|arXiv (Cornell University)|Mar 17, 2017

Reinforcement Learning in Robotics39 references188 citations

TL;DR

The paper formalizes multi-task multi-agent reinforcement learning under partial observability and proposes a two-phase decentralized approach: single-task learning with concurrent experience replay and hysteretic DRQNs, followed by distillation into a unified multi-task policy. It demonstrates robustness and coordination without explicit task identities.

ABSTRACT

Many real-world tasks involve multiple agents with partial observability and limited communication. Learning is challenging in these settings due to local viewpoints of agents, which perceive the world as non-stationary due to concurrently-exploring teammates. Approaches that learn specialized policies for individual tasks face problems when applied to the real world: not only do agents have to learn and store distinct policies for each task, but in practice identities of tasks are often non-observable, making these approaches inapplicable. This paper formalizes and addresses the problem of multi-task multi-agent reinforcement learning under partial observability. We introduce a decentralized single-task learning approach that is robust to concurrent interactions of teammates, and present an approach for distilling single-task policies into a unified policy that performs well across multiple related tasks, without explicit provision of task identity.

Motivation & Objective

• Formalize MT-MARL under partial observability with decentralized, independent learners.
• Develop a stable single-task MARL method robust to non-stationarity from teammates.
• Introduce Concurrent Experience Replay Trajectories (CERTs) for synchronized experience sharing in MARL.
• Distill specialized single-task policies into a unified multi-task policy without explicit task identity.
• Demonstrate effectiveness on multi-agent target capture domains across varying task settings.

Proposed method

• Use Decentralized Hysteretic Deep Recurrent Q-Networks (Dec-HDRQNs) to learn Q-values in partially observable MARL with hysteretic learning rates.
• Introduce Concurrent Experience Replay Trajectories (CERTs) to synchronize episode-wide experiences across agents for stable learning.
• Train DRQNs with sequential traces and backward-timestep sampling to stabilize learning under partial observability.
• Phase II distills multiple task-specific DRQNs into a single distilled DRQN by supervised regression over Q-values using a tempered KL-divergence loss.
• During distillation, use concurrent CERTs for storing regression experiences, but training is supervised rather than concurrent sampling.
• Formulate the MT-MARL problem with tasks drawn from a shared Dec-POMDP domain where task IDs are given to learners during training but not during execution.

Experimental results

Research questions

• RQ1 Can decentralized, independent learners achieve coordinated policies in MT-MARL under partial observability without explicit task identity?
• RQ2 Does hysteretic learning improve stability and coordination in Dec-POMDP MARL with non-stationarity from teammates?
• RQ3 Can single-task specialized policies be distilled into a unified multi-task policy that generalizes across related tasks?
• RQ4 What role do synchronized experience samples (CERTs) play in sample efficiency and stability in multi-agent learning?
• RQ5 How does the proposed MT-MARL approach perform on multi-agent target capture domains with varying task configurations?

Key findings

• Dec-HDRQN enables stable coordination in multi-agent partial observability where non-hysteretic methods fail.
• CERTs provide synchronized experiences that improve sample efficiency and stability in decentralized MARL.
• Distilled multi-task DRQNs closely match specialized task performance, enabling a unified policy without explicit task IDs at execution.
• Phase I specialization combined with Phase II distillation yields robust MT-MARL performance across varying grid sizes and task assignments.
• Hysteretic learning rates help mitigate negative transfer and destabilization due to teammates’ exploration in cooperative MARL.
• Centralized inter-agent training with parameter sharing did not improve performance compared to fully decentralized hysteretic learning.