[Paper Review] Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

Ryan Lowe|arXiv (Cornell University)|Jun 7, 2017

Reinforcement Learning in Robotics38 references1,014 citations

TL;DR

Introduces MADDPG, an actor-critic method with centralized critics that condition on other agents’ actions during training and decentralized execution, improving learning in cooperative, competitive, and mixed multi-agent tasks. It also uses policy ensembles and online modeling of others to boost robustness.

ABSTRACT

We explore deep reinforcement learning methods for multi-agent domains. We begin by analyzing the difficulty of traditional algorithms in the multi-agent case: Q-learning is challenged by an inherent non-stationarity of the environment, while policy gradient suffers from a variance that increases as the number of agents grows. We then present an adaptation of actor-critic methods that considers action policies of other agents and is able to successfully learn policies that require complex multi-agent coordination. Additionally, we introduce a training regimen utilizing an ensemble of policies for each agent that leads to more robust multi-agent policies. We show the strength of our approach compared to existing methods in cooperative as well as competitive scenarios, where agent populations are able to discover various physical and informational coordination strategies.

Motivation & Objective

• Motivate and analyze the challenges of applying traditional RL (Q-learning and policy gradient) in multi-agent settings due to non-stationarity and high gradient variance.
• Propose a general multi-agent deep reinforcement learning algorithm with centralized training and decentralized execution.
• Enable learning with local execution but centralized critics that access other agents' policies during training.
• Improve stability and robustness by modeling other agents online and by using ensembles of policies.

Proposed method

• Extend actor-critic policy gradients with a centralized critic that conditions on all agents' actions.
• Derive the gradient for agent i using a centralized Q^{pi}_i that takes as input all agents’ actions and some state information.
• Allow decentralized execution where each agent uses only local observations.
• Optionally learn approximations of other agents’ policies to relax the need for exact policy knowledge.
• Introduce policy ensembles to train multiple sub-policies per agent for robustness.

Experimental results

Research questions

• RQ1 Can centralized critics leveraging other agents’ actions stabilize learning in multi-agent environments with local execution?
• RQ2 Does modeling or approximating other agents’ policies during training improve performance when exact policies are unknown?
• RQ3 Do ensembles of sub-policies yield more robust multi-agent policies against non-stationarity and adversarial behavior?
• RQ4 How does MADDPG compare to single-agent DDPG in cooperative and competitive tasks?
• RQ5 What are the benefits and limitations of extending policy gradient to multi-agent contexts with centralized critics?

Key findings

• MADDPG outperforms DDPG and other baselines in both cooperative and competitive environments.
• Agents trained with MADDPG learn coordinated behaviors that single-agent approaches struggle to achieve.
• Using approximations of other agents’ policies yields comparable performance without slowing convergence.
• Policy ensembles provide stronger and more robust policies than single-policy agents in adversarial settings.
• Cooperative and deception tasks show MADDPG achieving higher success rates and lower adversary success in several scenarios.