[Paper Review] Is multiagent deep reinforcement learning the answer or the question? A brief survey

Pablo Hernández-Leal, Bilal Kartal+1 more|arXiv (Cornell University)|Oct 12, 2018

Reinforcement Learning in Robotics232 references58 citations

TL;DR

This paper surveys multiagent deep reinforcement learning (MDRL), reviewing key components from single-agent RL and MAL, offering practical guidelines for new researchers, and critically analyzing implementation and computational challenges. It aims to unify and advance the MDRL field by synthesizing existing literature and identifying open research directions.

ABSTRACT

Deep reinforcement learning (RL) has achieved outstanding results in recent years. This has led to a dramatic increase in the number of applications and methods. Recent works have explored learning beyond single-agent scenarios and have considered multiagent learning (MAL) scenarios. Initial results report successes in complex multiagent domains, although there are several challenges to be addressed. The primary goal of this article is to provide a clear overview of current multiagent deep reinforcement learning (MDRL) literature. Additionally, we complement the overview with a broader analysis: (i) we revisit previous key components, originally presented in MAL and RL, and highlight how they have been adapted to multiagent deep reinforcement learning settings. (ii) We provide general guidelines to new practitioners in the area: describing lessons learned from MDRL works, pointing to recent benchmarks, and outlining open avenues of research. (iii) We take a more critical tone raising practical challenges of MDRL (e.g., implementation and computational demands). We expect this article will help unify and motivate future research to take advantage of the abundant literature that exists (e.g., RL and MAL) in a joint effort to promote fruitful research in the multiagent community.

Motivation & Objective

• To provide a comprehensive overview of current multiagent deep reinforcement learning (MDRL) literature.
• To revisit and adapt foundational components from single-agent RL and multiagent learning (MAL) for MDRL settings.
• To offer practical guidelines for new practitioners, including lessons learned, recent benchmarks, and open research avenues.
• To critically assess practical challenges in MDRL, such as implementation complexity and computational demands.
• To unify and motivate future research by leveraging existing RL and MAL knowledge in a joint effort for the multiagent community.

Proposed method

• Systematically reviews and categorizes recent MDRL works across complex multiagent domains.
• Adapts key components from traditional RL and MAL—such as value function approximation, credit assignment, and policy gradient methods—for multiagent deep learning settings.
• Analyzes the evolution and integration of algorithms like MADQN, independent DQNs, and multiagent actor-critic methods.
• Evaluates benchmark environments used in MDRL, such as Hanabi, StarCraft Multi-Agent Challenge, and multi-robot navigation tasks.
• Identifies recurring design patterns and implementation pitfalls through critical analysis of published MDRL methods.
• Proposes a framework for evaluating MDRL approaches based on scalability, stability, and sample efficiency.

Experimental results

Research questions

• RQ1 How have core RL and MAL components been adapted for multiagent deep reinforcement learning?
• RQ2 What are the key challenges in implementing and scaling MDRL systems in practice?
• RQ3 Which benchmarks and evaluation protocols are most effective for assessing MDRL performance?
• RQ4 What lessons can new practitioners learn from existing MDRL literature to avoid common pitfalls?
• RQ5 What are the most promising open research directions in MDRL that could lead to scalable and stable multiagent systems?

Key findings

• MDRL has shown success in complex multiagent domains, but scalability and stability remain significant hurdles.
• Implementation complexity and high computational demands are major barriers to widespread adoption of MDRL methods.
• Recent benchmarks such as the StarCraft Multi-Agent Challenge and Hanabi provide valuable testbeds for evaluating MDRL algorithms.
• Independent deep Q-networks (DQNs) and multiagent actor-critic methods show promise but often suffer from policy divergence and non-stationarity.
• There is a lack of standardized evaluation protocols, making cross-method comparison difficult.
• The integration of insights from single-agent RL and MAL is essential for advancing robust and generalizable MDRL systems.