[Paper Review] Parameter Sharing Deep Deterministic Policy Gradient for Cooperative Multi-agent Reinforcement Learning

Xiangxiang Chu, Hangjun Ye|arXiv (Cornell University)|Oct 1, 2017

Reinforcement Learning in Robotics7 references54 citations

TL;DR

This paper introduces parameter sharing Deep Deterministic Policy Gradient variants to scale cooperative multi-agent reinforcement learning under local observations, improving learning speed and memory efficiency.

ABSTRACT

Deep reinforcement learning for multi-agent cooperation and competition has been a hot topic recently. This paper focuses on cooperative multi-agent problem based on actor-critic methods under local observations settings. Multi agent deep deterministic policy gradient obtained state of art results for some multi-agent games, whereas, it cannot scale well with growing amount of agents. In order to boost scalability, we propose a parameter sharing deterministic policy gradient method with three variants based on neural networks, including actor-critic sharing, actor sharing and actor sharing with partially shared critic. Benchmarks from rllab show that the proposed method has advantages in learning speed and memory efficiency, well scales with growing amount of agents, and moreover, it can make full use of reward sharing and exchangeability if possible.

Motivation & Objective

• Motivate cooperative multi-agent RL under local observations and address scalability with growing agent counts.
• Develop parameter-sharing variants of DDPG to improve training efficiency and memory usage.
• Showcase advantages of sharing mechanisms for reward sharing and exchangeability when applicable.

Proposed method

• Proposes three variants: actor-critic sharing, actor sharing, and actor sharing with partially shared critic.
• Bases innovations on the Deep Deterministic Policy Gradient framework.
• Uses neural networks to implement the sharing strategies.
• Evaluates on benchmarks from rllab to assess learning speed, memory efficiency, and scalability.

Experimental results

Research questions

• RQ1 How does parameter sharing affect learning speed in cooperative multi-agent RL under local observations?
• RQ2 How does memory usage scale with the number of agents for the proposed variants?
• RQ3 Do the proposed sharing schemes enable effective use of reward sharing and exchangeability when possible?
• RQ4 Which variant (actor-critic sharing, actor sharing, or partially shared critic) offers the best trade-off between performance and scalability?

Key findings

• Parameter sharing variants show advantages in learning speed and memory efficiency.
• Methods scale better as the number of agents grows.
• The approaches can leverage reward sharing and exchangeability when feasible.