[Paper Review] Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning

Georgios Papoudakis, Filippos Christianos+2 more|arXiv (Cornell University)|Jun 11, 2019

Reinforcement Learning in RoboticsComputer Science36 references124 citations

TL;DR

The paper surveys how non-stationarity arises in multi-agent deep RL and categorizes methods to mitigate it, including centralized critics, decentralized learning, opponent modeling, meta-learning, and communication, with open problems and future directions.

ABSTRACT

Recent developments in deep reinforcement learning are concerned with creating decision-making agents which can perform well in various complex domains. A particular approach which has received increasing attention is multi-agent reinforcement learning, in which multiple agents learn concurrently to coordinate their actions. In such multi-agent environments, additional learning problems arise due to the continually changing decision-making policies of agents. This paper surveys recent works that address the non-stationarity problem in multi-agent deep reinforcement learning. The surveyed methods range from modifications in the training procedure, such as centralized training, to learning representations of the opponent's policy, meta-learning, communication, and decentralized learning. The survey concludes with a list of open problems and possible lines of future research.

Motivation & Objective

• Motivate and define non-stationarity in multi-agent DRL and its impact on learning stability.
• Survey and categorize recent approaches addressing non-stationarity across training architectures and information assumptions.
• Identify promising directions and open problems for future research in multi-agent non-stationarity.

Proposed method

• Review and categorize existing methods for non-stationarity in multi-agent DRL.
• Provide taxonomy detailing training/execution architecture, modeling, opponent information, and algorithms.
• Summarize representative algorithms and their empirical settings in a consolidated table.

Experimental results

Research questions

• RQ1 What approaches have been proposed to address non-stationarity in multi-agent deep reinforcement learning?
• RQ2 How do centralized vs decentralized training, opponent modeling, meta-learning, learning representations, and communication contribute to stabilizing learning under non-stationarity?
• RQ3 What are the open problems and future research directions in this area?

Key findings

• Centralized critics with decentralized actors stabilize training by conditioning policy gradients on joint observations/actions.
• Opponent modeling and learning representations can mitigate non-stationarity and improve generalization to diverse opponents.
• Meta-learning approaches (e.g., MAML-inspired) enable rapid adaptation to non-stationary dynamics.
• Self-play and stabilized experience replay are effective decentralized strategies under non-stationarity.
• Communication among agents emerges as a useful mechanism to coordinate policies and stabilize learning in multi-agent settings.