[Paper Review] Off-Policy Multi-Agent Decomposed Policy Gradients

Yihan Wang, Beining Han+3 more|arXiv (Cornell University)|Jul 24, 2020

Reinforcement Learning in Robotics59 references42 citations

TL;DR

DOP introduces a decomposed, off-policy MAPG framework with a linearly factorized critic to enable efficient off-policy learning, mitigate centralized-decentralized mismatch, and improve credit assignment in both discrete and continuous action spaces, showing strong empirical gains on StarCraft II and multi-agent environments.

ABSTRACT

Multi-agent policy gradient (MAPG) methods recently witness vigorous progress. However, there is a significant performance discrepancy between MAPG methods and state-of-the-art multi-agent value-based approaches. In this paper, we investigate causes that hinder the performance of MAPG algorithms and present a multi-agent decomposed policy gradient method (DOP). This method introduces the idea of value function decomposition into the multi-agent actor-critic framework. Based on this idea, DOP supports efficient off-policy learning and addresses the issue of centralized-decentralized mismatch and credit assignment in both discrete and continuous action spaces. We formally show that DOP critics have sufficient representational capability to guarantee convergence. In addition, empirical evaluations on the StarCraft II micromanagement benchmark and multi-agent particle environments demonstrate that DOP significantly outperforms both state-of-the-art value-based and policy-based multi-agent reinforcement learning algorithms. Demonstrative videos are available at https://sites.google.com/view/dop-mapg/.

Motivation & Objective

• Diagnose why state-of-the-art multi-agent policy gradient methods underperform value-based methods.
• Propose a decomposed critic architecture to enable off-policy MAPG and address CDM and credit assignment.
• Establish theoretical convergence properties for decomposed critics.
• Demonstrate empirical effectiveness on discrete (StarCraft II SMAC) and continuous (multi-agent particle) benchmarks.

Proposed method

• Propose a centralized but factorized critic: Q_tot^φ(τ, a) = sum_i k_i(τ) Q_i^φ_i(τ, a_i) + b(τ).
• Enable tractable off-policy evaluation by reducing E_π[Q_tot^φ(τ, ·)] to a sum of agent-wise expectations.
• Combine decomposed tree backup with on-policy TD(λ) updates to balance sample and training efficiency.
• Provide stochastic and deterministic DOP variants for discrete and continuous action spaces, with corresponding policy gradient forms.
• Prove that decomposed critics preserve policy improvement under mild assumptions and show bias-variance trade-offs.
• Describe implementation details and network architecture in appendices.

Experimental results

Research questions

• RQ1 Does the centralized-decentralized mismatch (CDM) occur in practice, and can decomposed critics attenuate it?
• RQ2 Can off-policy stochastic MAPG methods be made tractable and stable with decomposed critics?
• RQ3 Can decomposed critics provide effective multi-agent credit assignment for deterministic policies?
• RQ4 Do DOP methods outperform state-of-the-art MAPG and value-based MARL methods on standard benchmarks?

Key findings

• Stochastic DOP significantly outperforms baselines on discrete-action StarCraft II micromanagement tasks.
• DOP reduces gradient variance and attenuates CDM through decomposed critic structure.
• Off-policy decomposed tree backup (TB) enables tractable multi-agent off-policy evaluation.
• Deterministic DOP learns credit assignment in continuous-action tasks and outperforms previous MAPG methods.
• DOP demonstrates stability and strong performance across multiple random seeds on challenging benchmarks.