[Paper Review] A Review of Cooperation in Multi-agent Learning

Yali Du, Joel Z. Leibo+3 more|arXiv (Cornell University)|Dec 8, 2023

Evolutionary Game Theory and Cooperation15 citations

TL;DR

This survey reviews fundamental concepts, problem settings, and algorithms for cooperative multi-agent learning, covering team-based and mixed-motive settings, challenges, benchmarks, and open questions.

ABSTRACT

Cooperation in multi-agent learning (MAL) is a topic at the intersection of numerous disciplines, including game theory, economics, social sciences, and evolutionary biology. Research in this area aims to understand both how agents can coordinate effectively when goals are aligned and how they may cooperate in settings where gains from working together are possible but possibilities for conflict abound. In this paper we provide an overview of the fundamental concepts, problem settings and algorithms of multi-agent learning. This encompasses reinforcement learning, multi-agent sequential decision-making, challenges associated with multi-agent cooperation, and a comprehensive review of recent progress, along with an evaluation of relevant metrics. Finally we discuss open challenges in the field with the aim of inspiring new avenues for research.

Motivation & Objective

• Synthesize fundamental concepts in multi-agent learning (MAL) and cooperation across disciplines.
• Differentiate between team-based and mixed-motive MAL and their unique challenges.
• Survey algorithms, benchmarks, and evaluation metrics used to study cooperative MAL.
• Identify open challenges and directions to inspire future research in cooperative AI.

Proposed method

• Define single-agent RL foundations and extend to multi-agent Markov games and POMGs.
• Classify MAL into cooperative, competitive, and mixed-motive settings with formal definitions (MDP/Markov games).
• Review learning paradigms (value-based, policy-based, CTDE) and representative algorithms for team-based cooperation.
• Discuss credit assignment, generalization to novel partners, and social dilemmas in mixed-motive MAL.
• Summarize benchmarks, metrics, and evaluation approaches used to study MAL cooperation.

Figure 1: A taxonomy of multi-agent systems (Schelling, 1960 ) .

Experimental results

Research questions

• RQ1 What are the fundamental concepts and frameworks used to study cooperation in MAL?
• RQ2 How do team-based and mixed-motive MAL differ in objectives, incentives, and algorithmic approaches?
• RQ3 What are the central challenges (credit assignment, non-stationarity, generalization) hindering cooperation, and how are they addressed?
• RQ4 Which algorithms and learning paradigms are most effective for cooperative MARL under different reward structures?
• RQ5 What metrics and benchmarks best capture cooperative behavior and social welfare in MAL?

Key findings

• Cooperation in MAL is analyzed under team-based (shared reward) and mixed-motive (individual rewards, social dilemmas) frameworks.
• Non-stationarity and scalability are core challenges as the number of agents grows and policies co-adapt.
• Credit assignment and distinguishing individual contributions are difficult in team settings, especially with sparse/delayed rewards.
• CTDE, policy/ value-based methods, and counterfactual baselines are prominent approaches to enable scalable cooperative learning.
• Generalization to novel partners (ad hoc teamwork) and robustness to exploitation are active research areas in mixed-motive MAL.

Figure 2: For a population of size $n$ , the Schelling diagram shows the payoff for an $n+1$ agent choosing to either defect or cooperate. The dotted line shows the average reward of the population.