[Paper Review] Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges

Ziyuan Zhou, Guanjun Liu+1 more|arXiv (Cornell University)|May 17, 2023

Reinforcement Learning in Robotics245 references10 citations

TL;DR

This survey reviews MARL methods, applications, and visions for trustworthy MARL, highlighting safety, robustness, generalization, ethical constraints, and human interaction in human-machine systems.

ABSTRACT

Multi-agent reinforcement learning (MARL) is a widely used Artificial Intelligence (AI) technique. However, current studies and applications need to address its scalability, non-stationarity, and trustworthiness. This paper aims to review methods and applications and point out research trends and visionary prospects for the next decade. First, this paper summarizes the basic methods and application scenarios of MARL. Second, this paper outlines the corresponding research methods and their limitations on safety, robustness, generalization, and ethical constraints that need to be addressed in the practical applications of MARL. In particular, we believe that trustworthy MARL will become a hot research topic in the next decade. In addition, we suggest that considering human interaction is essential for the practical application of MARL in various societies. Therefore, this paper also analyzes the challenges while MARL is applied to human-machine interaction.

Motivation & Objective

• Summarize basic MARL methods and typical application scenarios.
• Outline safety, robustness, generalization, and ethical constraints in practical MARL.
• Highlight the importance of human interaction in MARL for real-world systems.
• Discuss challenges and prospects for trustworthy MARL in the next decade.

Proposed method

• Describe single-agent RL foundations and key equations (MDP, Q-learning, DQN, policy gradient, DPG).
• Present multi-agent formulations via stochastic games and joint Q/V functions.
• Discuss learning cooperation with CTDE, including VDN, QMIX, QTRAN, QPLEX, and attention-based/mean-field approaches.
• Discuss learning communication methods (reinforced and differentiable) and topology learning.
• Outline mean-field MARL and scalability strategies for large agent populations.
• Address human-in-the-loop considerations and four trust-related dimensions (safety, robustness, generalization, ethical constraints).

Figure 1. The outline of this survey

Experimental results

Research questions

• RQ1 What are the main MARL methodological families and their training paradigms?
• RQ2 What application domains have MARL been applied to, and with what methodological choices?
• RQ3 What are the key limitations for safety, robustness, generalization, and ethical constraints in MARL?
• RQ4 How can human interaction be integrated into MARL for practical human-machine systems?
• RQ5 What challenges and visionary prospects define trustworthy MARL for the coming decade?

Key findings

• Centralized training with decentralized execution (CTDE) is a dominant paradigm for cooperative MARL.
• Value-based methods (VDN, QMIX, QTRAN, QPLEX) and policy-based methods (MADDPG and variants with attention) address credit assignment and scalability in MARL.
• Learning communication (reinforced and differentiable) and graph-based/topology-aware approaches improve coordination in many-agent settings.
• Mean-field MARL offers scalability to large agent populations by approximating interactions with average effects or neighborhood-based observations.
• Applications span smart transportation, UAVs, intelligent information systems, manufacturing, and finance, illustrating MARL’s versatility and the need for human-in-the-loop considerations.

Figure 2. Categories of Safety in MARL