[Paper Review] Multi-agent Reinforcement Learning in Sequential Social Dilemmas

Joel Z. Leibo, Vinícius Zambaldi+3 more|arXiv (Cornell University)|Feb 10, 2017

Evolutionary Game Theory and CooperationSocial Sciences40 references274 citations

TL;DR

The paper defines Sequential Social Dilemmas (SSDs) as temporally extended Markov games, and studies how independent deep Q-learning agents learn cooperation or defection in two environments—Gathering and Wolfpack—showing how environment factors influence cooperative behavior and highlighting differences from MGSD models.

ABSTRACT

Matrix games like Prisoner's Dilemma have guided research on social dilemmas for decades. However, they necessarily treat the choice to cooperate or defect as an atomic action. In real-world social dilemmas these choices are temporally extended. Cooperativeness is a property that applies to policies, not elementary actions. We introduce sequential social dilemmas that share the mixed incentive structure of matrix game social dilemmas but also require agents to learn policies that implement their strategic intentions. We analyze the dynamics of policies learned by multiple self-interested independent learning agents, each using its own deep Q-network, on two Markov games we introduce here: 1. a fruit Gathering game and 2. a Wolfpack hunting game. We characterize how learned behavior in each domain changes as a function of environmental factors including resource abundance. Our experiments show how conflict can emerge from competition over shared resources and shed light on how the sequential nature of real world social dilemmas affects cooperation.

Motivation & Objective

• Introduce sequential social dilemmas (SSDs) to capture temporally extended cooperation/defection.
• Demonstrate that SSDs preserve the mixed incentives of MGSDs while requiring policy-level cooperation.
• Analyze how environmental factors (resource abundance, conflict costs) shape learned behaviors.
• Showcase how independent learning agents reveal different cooperative dynamics than MGSD models.

Proposed method

• Define SSDs as Markov games with partial observability where outcomes from cooperation/defection policies form an empirical payoff matrix.
• Use two 2-player, partially observable Markov games (Gathering and Wolfpack) to study emergent behavior.
• Apply independent deep Q-network (DQN) learners with epsilon-greedy exploration and replay buffers to learn policies.
• Compute empirical payoff matrices via empirical game-theoretic analysis (EGTA) by sampling cooperative and defecting policies.
• Manipulate environmental parameters (apple abundance, tag duration, capture radius, team reward) to observe effects on cooperation.
• Treat the other agent as part of the environment, avoiding prescriptive modeling of the other agent’s learning.

Experimental results

Research questions

• RQ1 How do environmental factors influence the emergence of cooperative versus defecting policies in SSDs?
• RQ2 Do SSDs yield qualitatively different dynamics and equilibria than MGSDs when learned via independent deep RL?
• RQ3 What heterogeneous cooperative strategies emerge under different resource and interaction costs?
• RQ4 How do agent architecture and learning parameters affect the propensity to defect or cooperate?

Key findings

• Environment scarcity and higher conflict costs promote more aggressive, defecting policies in Gathering.
• In Wolfpack, higher group benefit and larger capture radius increase cooperative, multi-agent hunting behavior.
• Empirical payoff matrices in these SSDs often reflect Prisoner’s Dilemma payoffs, but the SSD view reveals distinct game structures across Gathering and Wolfpack.
• Increased network size can increase cooperation in Wolfpack but increase defection in Gathering, showing task-dependent effects of cognitive capacity.
• SSD analysis uncovers coordination and implementation complexities not captured by MGSD models.