[Paper Review] Dealing With Non-stationarity in Decentralized Cooperative Multi-Agent Deep Reinforcement Learning via Multi-Timescale Learning

Hadi Nekoei, Akilesh Badrinaaraayanan+5 more|arXiv (Cornell University)|Feb 6, 2023

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes a multi-timescale learning algorithm for decentralized cooperative multi-agent reinforcement learning (MARL) to address non-stationarity caused by concurrent policy updates. By allowing agents to learn simultaneously at different rates—where the primary agent updates faster and others more slowly—the method accelerates training while reducing policy drift, outperforming state-of-the-art decentralized MARL methods on the ePymarl benchmark.

ABSTRACT

Decentralized cooperative multi-agent deep reinforcement learning (MARL) can be a versatile learning framework, particularly in scenarios where centralized training is either not possible or not practical. One of the critical challenges in decentralized deep MARL is the non-stationarity of the learning environment when multiple agents are learning concurrently. A commonly used and efficient scheme for decentralized MARL is independent learning in which agents concurrently update their policies independently of each other. We first show that independent learning does not always converge, while sequential learning where agents update their policies one after another in a sequence is guaranteed to converge to an agent-by-agent optimal solution. In sequential learning, when one agent updates its policy, all other agent's policies are kept fixed, alleviating the challenge of non-stationarity due to simultaneous updates in other agents' policies. However, it can be slow because only one agent is learning at any time. Therefore it might also not always be practical. In this work, we propose a decentralized cooperative MARL algorithm based on multi-timescale learning. In multi-timescale learning, all agents learn simultaneously, but at different learning rates. In our proposed method, when one agent updates its policy, other agents are allowed to update their policies as well, but at a slower rate. This speeds up sequential learning, while also minimizing non-stationarity caused by other agents updating concurrently. Multi-timescale learning outperforms state-of-the-art decentralized learning methods on a set of challenging multi-agent cooperative tasks in the epymarl(Papoudakis et al., 2020) benchmark. This can be seen as a first step towards more general decentralized cooperative deep MARL methods based on multi-timescale learning.

Motivation & Objective

• To address the challenge of non-stationarity in decentralized cooperative multi-agent reinforcement learning (MARL), where concurrent policy updates destabilize learning.
• To overcome the limitations of independent learning, which lacks convergence guarantees, and sequential learning, which is slow due to single-agent updates.
• To design a decentralized MARL algorithm that enables simultaneous learning across agents while minimizing policy drift from other agents' updates.
• To improve training efficiency and performance in cooperative MARL settings without requiring centralized training or coordination.
• To establish multi-timescale learning as a viable and effective paradigm for scalable, decentralized MARL.

Proposed method

• The method employs a multi-timescale learning framework where agents update their policies simultaneously but at different learning rates.
• The primary agent uses a faster learning rate, while other agents use slower learning rates to reduce the impact of their policy changes on the primary agent.
• This design allows for faster convergence than sequential learning while maintaining stability by minimizing non-stationarity from concurrent updates.
• The algorithm is fully decentralized, requiring no central coordination or shared parameter updates during training.
• The approach is implemented using independent deep Q-networks (DQN) or policy gradient methods, with learning rates explicitly scheduled per agent.
• The method is evaluated on cooperative tasks from the ePymarl benchmark, using standard MARL environments such as SMAC and Hanabi.

Experimental results

Research questions

• RQ1 Can a decentralized MARL algorithm achieve faster convergence than sequential learning while maintaining stability?
• RQ2 Does multi-timescale learning reduce non-stationarity compared to independent learning in cooperative MARL?
• RQ3 Can simultaneous learning at different rates outperform existing state-of-the-art decentralized MARL methods in cooperative environments?
• RQ4 How does the choice of learning rate ratios affect learning stability and sample efficiency in multi-agent settings?
• RQ5 Is multi-timescale learning a scalable and generalizable framework for decentralized cooperative MARL?

Key findings

• The proposed multi-timescale learning method achieves faster convergence than sequential learning, which updates only one agent at a time.
• The method significantly reduces non-stationarity compared to independent learning, where all agents update simultaneously at similar rates.
• On the ePymarl benchmark, the method outperforms state-of-the-art decentralized MARL algorithms in terms of final return and training stability.
• The performance gain is particularly evident in complex cooperative environments such as SMAC (Simplified Multi-Agent Control) and Hanabi.
• The use of slower learning rates for secondary agents effectively dampens policy drift, improving overall learning stability.
• Empirical results show that the method maintains high sample efficiency and generalization across diverse cooperative tasks.