[Paper Review] Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing

Filippos Christianos, Georgios Papoudakis+2 more|arXiv (Cornell University)|Feb 15, 2021

Reinforcement Learning in Robotics24 references17 citations

TL;DR

This paper proposes Selective Parameter Sharing (SePS), a method that dynamically partitions agents into clusters based on their learned behaviors and goals before training, enabling efficient and effective parameter sharing only among similar agents. SePS improves sample efficiency and training speed while achieving higher final returns than naive parameter sharing or independent networks, scaling successfully to hundreds of heterogeneous agents in complex environments.

ABSTRACT

Sharing parameters in multi-agent deep reinforcement learning has played an essential role in allowing algorithms to scale to a large number of agents. Parameter sharing between agents significantly decreases the number of trainable parameters, shortening training times to tractable levels, and has been linked to more efficient learning. However, having all agents share the same parameters can also have a detrimental effect on learning. We demonstrate the impact of parameter sharing methods on training speed and converged returns, establishing that when applied indiscriminately, their effectiveness is highly dependent on the environment. We propose a novel method to automatically identify agents which may benefit from sharing parameters by partitioning them based on their abilities and goals. Our approach combines the increased sample efficiency of parameter sharing with the representational capacity of multiple independent networks to reduce training time and increase final returns.

Motivation & Objective

• To address the limitations of naive parameter sharing in multi-agent reinforcement learning, where indiscriminate sharing harms learning in heterogeneous environments.
• To improve training efficiency and final performance by identifying agent groups that can benefit from shared representations.
• To enable scaling MARL to hundreds of agents by combining the benefits of parameter sharing with the representational capacity of independent networks.
• To reduce interference between agents during training by preventing conflicting policy updates in shared parameters.
• To provide a pre-training method for architecture configuration that enhances learning without requiring dynamic reconfiguration during training.

Proposed method

• Agents are encoded into an embedding space using trajectories collected before policy training begins.
• An unsupervised clustering algorithm is applied to the embeddings to group agents with similar behaviors and goals.
• Parameter sharing is applied only within each cluster, creating separate neural network heads for different agent groups.
• The method operates as a pre-training step, establishing a fixed architecture before policy optimization begins.
• SePS is compatible with centralised training with experience sharing (CTDE) frameworks such as MADDPG, Q-Mix, and SEAC.
• The approach avoids the computational overhead of per-agent networks while maintaining representational capacity for diverse roles.

Experimental results

Research questions

• RQ1 Does naive parameter sharing degrade learning performance in heterogeneous multi-agent environments?
• RQ2 Can clustering agents based on behavioral similarity improve the effectiveness of parameter sharing in MARL?
• RQ3 To what extent does selective parameter sharing enhance training speed and final return compared to full sharing or no sharing?
• RQ4 Can SePS scale to hundreds of agents when agents have distinct roles and dynamics?
• RQ5 How does SePS compare to experience-sharing baselines like SEAC in terms of computational efficiency and performance?

Key findings

• SePS achieved higher final returns than both naive parameter sharing and independent networks across multiple environments.
• SePS reduced training time significantly compared to independent networks, while maintaining or improving sample efficiency.
• In environments with non-homogeneous agents, SePS outperformed full parameter sharing (FuPS) and no parameter sharing (NoPS) in both convergence speed and final performance.
• SePS successfully scaled to up to 200 agents in non-homogeneous environments, demonstrating robustness to large-scale multi-agent settings.
• The method effectively identified agent groupings based on behavioral similarity, enabling targeted parameter sharing without interference between dissimilar roles.
• Reinforcement learning with SePS showed reduced interference between agents during training, as agents no longer updated conflicting shared parameters.