[Paper Review] Parameter Sharing is Surprisingly Useful for Multi-Agent Deep Reinforcement Learning.

Justin K. Terry, Nathaniel Grammel+4 more|arXiv (Cornell University)|May 27, 2020

Reinforcement Learning in Robotics25 references29 citations

TL;DR

This paper demonstrates that parameter sharing—despite being underused in multi-agent deep reinforcement learning (MARL)—significantly improves learning efficiency and performance by reducing nonstationarity through centralized training. It shows that parameter sharing enables up to 44× higher average reward in just 16% of the episodes compared to prior setups, and provides a formal extension for use in heterogeneous environments.

ABSTRACT

Nonstationarity is a fundamental problem in cooperative multi-agent reinforcement learning (MARL)--each agent must relearn information about the other agent's policies due to the other agents learning, causing information to ring between agents and convergence to be slow. The MAILP model, introduced by Terry and Grammel (2020), is a novel model of information transfer during multi-agent learning. We use the MAILP model to show that increasing training centralization arbitrarily mitigates the slowing of convergence due to nonstationarity. The most centralized case of learning is parameter sharing, an uncommonly used MARL method, specific to environments with homogeneous agents, that bootstraps a single-agent reinforcement learning (RL) methods and learns an identical policy for each agent. We experimentally replicate the result of increased learning centralization leading to better performance on the MARL benchmark set from Gupta et al. (2017). We further apply parameter sharing to 8 modern single-agent deep RL (DRL) methods for the first time in the literature. With this, we achieved the best documented performance on a set of MARL benchmarks and achieved upto 44 times more average reward in as little as 16% as many episodes compared to documented parameter sharing arrangement. We finally offer a formal proof of a set of methods that allow parameter sharing to serve in environments with heterogeneous agents.

Motivation & Objective

• Address the challenge of nonstationarity in cooperative multi-agent reinforcement learning (MARL), where agents' policies change dynamically during training.
• Investigate whether increasing training centralization, particularly via parameter sharing, can mitigate convergence slowdowns caused by nonstationarity.
• Extend parameter sharing to heterogeneous multi-agent environments, where agents have different policies or observations.
• Evaluate parameter sharing across 8 modern deep RL algorithms on MARL benchmarks to establish state-of-the-art performance.
• Provide a formal proof for extending parameter sharing to heterogeneous agents, broadening its applicability.

Proposed method

• Adopt the MAILP model (Terry & Grammel, 2020) to formalize information transfer dynamics during multi-agent learning.
• Implement parameter sharing as the most centralized training regime, where all agents share a single policy network.
• Apply parameter sharing to 8 modern single-agent deep RL methods (e.g., SAC, PPO, DQN) for the first time in MARL.
• Train with shared parameters to bootstrap single-agent RL methods, enabling faster convergence and improved sample efficiency.
• Propose and formally prove a set of conditions under which parameter sharing can be adapted for heterogeneous agents.
• Use the MARL benchmark suite from Gupta et al. (2017) to evaluate performance across diverse cooperative environments.

Experimental results

Research questions

• RQ1 To what extent does increasing training centralization via parameter sharing reduce convergence delays caused by nonstationarity in MARL?
• RQ2 Can parameter sharing be successfully applied to modern deep RL algorithms in multi-agent settings, and does it improve sample efficiency?
• RQ3 What performance gains does parameter sharing achieve compared to prior MARL methods on standard benchmarks?
• RQ4 Is it possible to extend parameter sharing to heterogeneous multi-agent environments, and under what formal conditions does it remain effective?
• RQ5 How does parameter sharing compare in sample efficiency and final performance to non-shared, decentralized training approaches?

Key findings

• Parameter sharing achieved up to 44 times higher average cumulative reward compared to documented parameter-sharing setups in the same benchmarks.
• The method required as little as 16% of the number of episodes used in prior parameter-sharing experiments to achieve superior performance.
• The performance gains were consistent across the MARL benchmark suite from Gupta et al. (2017), confirming the effectiveness of increased centralization.
• The study established the best documented performance on the benchmark set, setting a new state-of-the-art for MARL with parameter sharing.
• A formal proof was provided for extending parameter sharing to heterogeneous agents, enabling its use in previously restricted environments.
• The results confirm that parameter sharing is a highly effective and underutilized method for improving MARL training stability and sample efficiency.