[Paper Review] Deep reinforcement learning models the emergent dynamics of human cooperation

Kevin R. McKee, Edward Hughes+8 more|arXiv (Cornell University)|Mar 8, 2021

Evolutionary Game Theory and Cooperation2 references4 citations

TL;DR

This paper uses multi-agent deep reinforcement learning to model how intrinsic reputation motivation drives human-like spatial and temporal coordination in collective action. The model successfully predicts that reputation-seeking leads groups to adopt a non-territorial, turn-taking strategy in a public goods dilemma, aligning with observed human behavioral data.

ABSTRACT

Collective action demands that individuals efficiently coordinate how much, where, and when to cooperate. Laboratory experiments have extensively explored the first part of this process, demonstrating that a variety of social-cognitive mechanisms influence how much individuals choose to invest in group efforts. However, experimental research has been unable to shed light on how social cognitive mechanisms contribute to the where and when of collective action. We leverage multi-agent deep reinforcement learning to model how a social-cognitive mechanism--specifically, the intrinsic motivation to achieve a good reputation--steers group behavior toward specific spatial and temporal strategies for collective action in a social dilemma. We also collect behavioral data from groups of human participants challenged with the same dilemma. The model accurately predicts spatial and temporal patterns of group behavior: in this public goods dilemma, the intrinsic motivation for reputation catalyzes the development of a non-territorial, turn-taking strategy to coordinate collective action.

Motivation & Objective

• To investigate how social-cognitive mechanisms, particularly reputation motivation, influence the timing and spatial coordination of collective action.
• To address the gap in experimental research on 'where' and 'when' cooperation emerges, beyond just 'how much' individuals contribute.
• To develop a computational model that captures emergent coordination patterns in group dilemmas using deep reinforcement learning.
• To validate the model against empirical behavioral data from human participants in a controlled social dilemma.
• To demonstrate that reputation motivation leads to non-territorial, turn-taking strategies in group cooperation.

Proposed method

• Employed multi-agent deep reinforcement learning to simulate groups of agents learning to coordinate cooperation in a public goods dilemma.
• Integrated intrinsic reputation as a reward signal, where agents gain rewards for maintaining a high reputation among peers.
• Trained agents using deep Q-networks (DQN) with shared policy networks to enable emergent coordination without explicit rules.
• Simulated spatial and temporal dynamics by modeling agents' positions and turn-taking behavior in a structured environment.
• Calibrated the model using behavioral data from human experiments to ensure ecological validity.
• Evaluated model performance by comparing predicted spatial and temporal coordination patterns against observed human behavior.

Experimental results

Research questions

• RQ1 How does intrinsic reputation motivation shape the timing and spatial distribution of cooperation in a group dilemma?
• RQ2 Can deep reinforcement learning models replicate the emergent coordination patterns observed in human groups?
• RQ3 Does reputation-driven behavior lead to non-territorial, turn-taking strategies in collective action?
• RQ4 How do social-cognitive mechanisms like reputation influence the emergence of structured cooperation beyond simple contribution levels?
• RQ5 To what extent does the model's predicted behavior match real human behavioral data in a public goods setting?

Key findings

• The model accurately predicted the emergence of non-territorial, turn-taking strategies in group cooperation, matching observed human behavior.
• Reputation motivation alone was sufficient to drive coordinated, spatially distributed cooperation without explicit communication or rules.
• Human participants exhibited similar turn-taking and non-territorial coordination patterns, validating the model's predictions.
• The model demonstrated that reputation-based incentives can lead to efficient, self-organized coordination in social dilemmas.
• Spatial and temporal coordination emerged organically from reputation-driven learning, highlighting the role of social-cognitive mechanisms in collective dynamics.
• The alignment between model predictions and human data suggests that reputation is a key driver of emergent coordination in real-world collective action.