[Paper Review] Selectively Sharing Experiences Improves Multi-Agent Reinforcement Learning

Matthias Gerstgrasser, Tom Danino+1 more|arXiv (Cornell University)|Nov 1, 2023

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper introduces SUPER (Selective Multi-Agent Prioritized Experience Relay), a decentralized multi-agent reinforcement learning framework that improves learning by selectively sharing only the highest-temporal-difference (TD) error experiences among agents. By prioritizing and relaying only the most informative transitions—using TD error as a proxy for relevance—SUPER achieves faster convergence and superior performance over both non-sharing DQN and state-of-the-art multi-agent algorithms, even with minimal communication bandwidth.

ABSTRACT

We present a novel multi-agent RL approach, Selective Multi-Agent Prioritized Experience Relay, in which agents share with other agents a limited number of transitions they observe during training. The intuition behind this is that even a small number of relevant experiences from other agents could help each agent learn. Unlike many other multi-agent RL algorithms, this approach allows for largely decentralized training, requiring only a limited communication channel between agents. We show that our approach outperforms baseline no-sharing decentralized training and state-of-the art multi-agent RL algorithms. Further, sharing only a small number of highly relevant experiences outperforms sharing all experiences between agents, and the performance uplift from selective experience sharing is robust across a range of hyperparameters and DQN variants. A reference implementation of our algorithm is available at https://github.com/mgerstgrasser/super.

Motivation & Objective

• To address the challenge of non-stationarity and high variance in multi-agent reinforcement learning by leveraging inter-agent experience sharing.
• To improve sample efficiency and learning speed in decentralized training settings without requiring centralized training infrastructure.
• To investigate whether selective, prioritized experience sharing—based on TD error—can outperform indiscriminate or uniform experience sharing in multi-agent DQN environments.
• To develop a communication-efficient, semi-decentralized training paradigm that maintains agent autonomy while enabling collaborative learning.

Proposed method

• Agents independently train using DQN with their own experience replay buffers.
• Each agent computes the temporal difference (TD) error for every transition in its replay buffer to assess learning uncertainty.
• Only the top-k experiences with the highest TD errors are selected for sharing with other agents.
• Shared experiences are inserted directly into the receiving agents’ replay buffers, enabling off-policy learning with prioritized data.
• The approach is compatible with or without prior use of prioritized experience replay (PER) in individual agents.
• Communication is limited to only the most relevant experiences, minimizing bandwidth while maximizing learning impact.

Experimental results

Research questions

• RQ1 Can selective sharing of high-impact experiences improve multi-agent RL performance compared to no sharing or full sharing?
• RQ2 Does using TD error as a heuristic for experience selection lead to better learning efficiency and final performance than uniform or random selection?
• RQ3 Can a decentralized training framework with minimal communication still achieve performance comparable to centralized training approaches?
• RQ4 How robust is the performance gain of selective experience sharing across different hyperparameters and DQN variants?
• RQ5 Does the use of deterministic vs. stochastic experience selection affect performance, and why?

Key findings

• SUPER outperforms non-sharing decentralized DQN training across all evaluated environments and hyperparameter settings.
• Selective sharing of only the highest-TD-error experiences significantly outperforms sharing all experiences, indicating that indiscriminate sharing is detrimental.
• The performance gain from selective experience sharing is robust across different DQN variants and hyperparameter configurations.
• Deterministic selection of top experiences (rather than stochastic) yields slightly better results, likely due to the one-time nature of sharing and reduced risk of missing critical experiences.
• The method achieves strong performance even with very low communication bandwidth, demonstrating feasibility in resource-constrained settings.
• SUPER improves performance over state-of-the-art algorithms like MADDPG and SEAC, particularly in sparse-reward environments.