[Paper Review] ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependency

Chuming Li, Jie Liu+6 more|arXiv (Cornell University)|Nov 29, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes ACE, a cooperative multi-agent reinforcement learning algorithm that resolves the non-stationarity problem via bidirectional action-dependency, transforming a multi-agent MDP into a single-agent MDP through sequential decision-making. ACE achieves state-of-the-art performance, attaining 100% win rates on nearly all hard and super-hard maps in SMAC and significantly outperforming prior methods in sample efficiency and cumulative return on GRF and SMAC benchmarks.

ABSTRACT

Multi-agent reinforcement learning (MARL) suffers from the non-stationarity problem, which is the ever-changing targets at every iteration when multiple agents update their policies at the same time. Starting from first principle, in this paper, we manage to solve the non-stationarity problem by proposing bidirectional action-dependent Q-learning (ACE). Central to the development of ACE is the sequential decision-making process wherein only one agent is allowed to take action at one time. Within this process, each agent maximizes its value function given the actions taken by the preceding agents at the inference stage. In the learning phase, each agent minimizes the TD error that is dependent on how the subsequent agents have reacted to their chosen action. Given the design of bidirectional dependency, ACE effectively turns a multiagent MDP into a single-agent MDP. We implement the ACE framework by identifying the proper network representation to formulate the action dependency, so that the sequential decision process is computed implicitly in one forward pass. To validate ACE, we compare it with strong baselines on two MARL benchmarks. Empirical experiments demonstrate that ACE outperforms the state-of-the-art algorithms on Google Research Football and StarCraft Multi-Agent Challenge by a large margin. In particular, on SMAC tasks, ACE achieves 100% success rate on almost all the hard and super-hard maps. We further study extensive research problems regarding ACE, including extension, generalization, and practicability. Code is made available to facilitate further research.

Motivation & Objective

• To address the non-stationarity problem in cooperative multi-agent reinforcement learning (MARL), where changing policies of agents disrupt value function learning.
• To develop a method that enables accurate estimation of individual action values despite joint action space complexity and policy co-evolution.
• To transform a cooperative multi-agent MDP into a single-agent MDP using sequential decision-making with bidirectional dependencies.
• To improve sample efficiency, convergence, and generalization in MARL benchmarks like Google Research Football and StarCraft Multi-Agent Challenge.
• To enable practical deployment via integration with the CTDE framework and support for decentralized execution.

Proposed method

• ACE formulates a sequentially expanded MDP (SE-MDP) where each agent acts in sequence, with state expansions based on prior actions, effectively turning MMDP into a single-agent MDP.
• It introduces bidirectional action-dependency: forward dependency (value depends on prior actions) and backward dependency (target depends on subsequent agent reactions).
• The SE-state is represented via additive combination of unit embeddings and action embeddings, enabling efficient computation of all intermediate states in one forward pass.
• An interaction-aware action embedding is introduced to model inter-agent interactions, improving value estimation and policy performance.
• The method is compatible with standard Q-learning and actor-critic frameworks, such as PPO, via distillation of joint action values into individual value functions.
• A counterfactual distillation technique is used in ACE-CTDE to transfer joint action knowledge into decentralized individual policies, preserving the IGM property.

Experimental results

Research questions

• RQ1 Can bidirectional action-dependency effectively resolve the non-stationarity problem in cooperative MARL by transforming MMDP into a single-agent MDP?
• RQ2 How does ACE compare in performance and sample efficiency to state-of-the-art MARL algorithms like QMIX and MAPPO on complex benchmarks?
• RQ3 To what extent does ACE generalize to new tasks with different numbers of agents, especially when trained on a fixed configuration?
• RQ4 How robust is ACE to the order of agent decision-making in the sequential process?
• RQ5 Can ACE be effectively adapted to the CTDE paradigm while maintaining high performance under decentralized execution?

Key findings

• ACE achieves 100% win rates on almost all hard and super-hard maps in the StarCraft Multi-Agent Challenge (SMAC), significantly outperforming existing methods.
• On Google Research Football, ACE achieves superior cumulative returns and improves sample efficiency by up to 500% compared to state-of-the-art algorithms.
• ACE demonstrates strong generalization: it maintains high performance when transferred to new maps with different numbers of agents without fine-tuning.
• The interaction-aware action embedding contributes significantly to performance, as shown by the drop in performance when removed (ACE-w/o-IA vs. ACE).
• ACE is robust to agent ordering, with minimal performance difference between shuffled and sorted decision sequences.
• ACE-CTDE achieves near-identical performance to full ACE in decentralized execution, validating the effectiveness of counterfactual distillation for maintaining IGM property.