[Paper Review] MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learning

Kefan Su, Siyuan Zhou+4 more|arXiv (Cornell University)|Sep 17, 2022

Reinforcement Learning in Robotics5 citations

TL;DR

MA2QL proposes a minimalist, fully decentralized multi-agent reinforcement learning algorithm where agents take turns updating their Q-functions via Q-learning to address non-stationarity. Despite minimal changes to independent Q-learning (IQL), MA2QL achieves consistent performance gains across diverse environments and theoretically guarantees convergence to a Nash equilibrium under $\varepsilon$-convergence conditions.

ABSTRACT

Decentralized learning has shown great promise for cooperative multi-agent reinforcement learning (MARL). However, non-stationarity remains a significant challenge in fully decentralized learning. In the paper, we tackle the non-stationarity problem in the simplest and fundamental way and propose multi-agent alternate Q-learning (MA2QL), where agents take turns updating their Q-functions by Q-learning. MA2QL is a minimalist approach to fully decentralized cooperative MARL but is theoretically grounded. We prove that when each agent guarantees $\varepsilon$-convergence at each turn, their joint policy converges to a Nash equilibrium. In practice, MA2QL only requires minimal changes to independent Q-learning (IQL). We empirically evaluate MA2QL on a variety of cooperative multi-agent tasks. Results show MA2QL consistently outperforms IQL, which verifies the effectiveness of MA2QL, despite such minimal changes.

Motivation & Objective

• To address the non-stationarity problem in fully decentralized cooperative multi-agent reinforcement learning (MARL).
• To develop a minimal yet theoretically grounded approach that requires only minor modifications to independent Q-learning (IQL).
• To empirically verify that turn-based learning stabilizes training and improves performance in complex, partially observable environments.
• To demonstrate scalability and robustness of the method across discrete and continuous action spaces, and fully/partially observable settings.

Proposed method

• Agents alternate in updating their Q-functions using standard Q-learning, while all agents interact with the environment during each update turn.
• The method enforces a fixed learning schedule where each agent updates in sequence, effectively decoupling policy updates to reduce non-stationarity.
• Theoretical analysis proves that if each agent achieves $\varepsilon$-convergence during its update turn, the joint policy converges to a Nash equilibrium.
• The algorithm is implemented with minimal code changes: swapping the order of environment interaction and agent update in IQL's training loop.
• The approach is compatible with both discrete-action (e.g., MPE, SMAC) and continuous-action (e.g., multi-agent MuJoCo) environments using DDPG.
• No global state or communication is used during training, preserving full decentralization.

Experimental results

Research questions

• RQ1 Can a minimal, turn-based update schedule stabilize learning in fully decentralized MARL and mitigate non-stationarity?
• RQ2 Does alternating Q-learning converge to a Nash equilibrium in tabular Dec-POMDP settings?
• RQ3 Can MA2QL outperform independent Q-learning (IQL) with only minor code changes across diverse MARL benchmarks?
• RQ4 How robust is MA2QL to hyperparameter variations and scaling to larger multi-agent environments?
• RQ5 Does the alternate learning schedule improve performance in partially observable and high-complexity environments?

Key findings

• In the stochastic game, MA2QL converges significantly faster than IQL, demonstrating improved stability and convergence speed.
• On the StarCraft Multi-Agent Challenge (SMAC), MA2QL outperforms IQL on hard and super-hard maps ($\mathtt{5m\_vs\_6m}$ and $\mathtt{corridor}$), achieving higher winning rates.
• In multi-agent MuJoCo, MA2QL consistently outperforms IQL across 15-agent and 17-agent tasks, showing strong scalability and robustness.
• Hyperparameter ablation shows that MA2QL maintains performance advantages over IQL across different learning rates and batch sizes, indicating robustness.
• In 3-agent Hopper and 3×2 HalfCheetah tasks, MA2QL achieves better final performance than IQL, even when K (number of updates per turn) varies.
• Theoretical analysis confirms that under $\varepsilon$-convergence per agent per turn, MA2QL converges to a Nash equilibrium in tabular settings.