[Paper Review] Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning

Jakob Foerster, Nantas Nardelli+5 more|arXiv (Cornell University)|Feb 28, 2017

Reinforcement Learning in Robotics29 references333 citations

TL;DR

This paper presents two methods to stabilize deep multi-agent Q-learning with experience replay: multi-agent importance sampling and a fingerprinting approach to disambiguate replay data from different training stages.

ABSTRACT

Many real-world problems, such as network packet routing and urban traffic control, are naturally modeled as multi-agent reinforcement learning (RL) problems. However, existing multi-agent RL methods typically scale poorly in the problem size. Therefore, a key challenge is to translate the success of deep learning on single-agent RL to the multi-agent setting. A major stumbling block is that independent Q-learning, the most popular multi-agent RL method, introduces nonstationarity that makes it incompatible with the experience replay memory on which deep Q-learning relies. This paper proposes two methods that address this problem: 1) using a multi-agent variant of importance sampling to naturally decay obsolete data and 2) conditioning each agent's value function on a fingerprint that disambiguates the age of the data sampled from the replay memory. Results on a challenging decentralised variant of StarCraft unit micromanagement confirm that these methods enable the successful combination of experience replay with multi-agent RL.

Motivation & Objective

• Motivate the challenge of applying deep RL with experience replay to fully cooperative multi-agent settings under nonstationarity.
• Develop methods that enable stable, sample-efficient learning when agents learn concurrently.
• Demonstrate the effectiveness of the proposed methods in a challenging, partially observable multi-agent domain ( decentralised StarCraft micromanagement ).
• Assess whether experience replay can be effectively combined with deep multi-agent Q-learning.
• Provide analysis of how nonstationarity affects replay and how to mitigate it.

Proposed method

• Introduce multi-agent importance sampling by augmenting replay tuples with the other agents’ policy information and applying importance-weighted loss during replay learning.
• Formulate an augmented (off-environment) Bellman equation to account for the changing policies of other agents and derive an importance sampling correction.
• Propose multi-agent fingerprints by conditioning each agent’s Q-function on low-dimensional indicators (e.g., training iteration e and exploration rate ε) to disambiguate replay data origins.
• Extend deep Q-learning with recurrent architectures to handle partial observability and evaluate both feed-forward and recurrent models.
• Conduct experiments in a decentralised StarCraft micromanagement domain to compare XP, XP+IS, and XP+FP variants with and without replay.
• Clip and normalise importance weights to control variance in the multi-agent setting.

Experimental results

Research questions

• RQ1 Can experience replay be stabilised in deep multi-agent Q-learning despite nonstationarity from concurrent learning agents?
• RQ2 Do importance sampling and data-age fingerprints effectively mitigate nonstationarity in replay data for multi-agent RL?
• RQ3 How do these methods affect learning stability and sample efficiency in a decentralised StarCraft micromanagement task?

Key findings

• Experience replay without stabilization (NOXP) performs poorly due to overfitting to greedy policies as exploration declines.
• Plain experience replay (XP) improves stability and performance over NOXP but cannot fully account for nonstationarity from other agents.
• Multi-agent importance sampling (XP+IS) provides slight gains with feed-forward models but is approximate in partially observable settings and can suffer high-variance weights.
• Fingerprints (XP+FP) dramatically improve performance for feed-forward models by disambiguating training stages, enabling better use of replay data; combining IS with FP does not yield additional benefits.
• Recurrent models show that trajectory information partially mitigates nonstationarity, with fingerprints helping still further by encoding stage-related information into observations.
• Overall, XP+FP consistently outperforms baselines in the tested decentralised StarCraft micromanagement tasks.