[Paper Review] Towards Cooperation in Sequential Prisoner's Dilemmas: a Deep Multiagent Reinforcement Learning Approach

Weixun Wang, Jianye Hao+2 more|arXiv (Cornell University)|Mar 1, 2018

Evolutionary Game Theory and CooperationSocial Sciences25 references16 citations

TL;DR

This paper proposes a deep multiagent reinforcement learning approach for achieving mutual cooperation in Sequential Prisoner’s Dilemma (SPD) games, where cooperation is temporally extended and graded. The method uses an offline phase to generate policies with varying cooperation levels and train a cooperation degree detection network via LSTM-based encoding, followed by an online phase that adaptively selects policies based on detected opponent cooperation, achieving robust cooperation while resisting exploitation in both self-play and dynamic opponent scenarios.

ABSTRACT

The Iterated Prisoner's Dilemma has guided research on social dilemmas for decades. However, it distinguishes between only two atomic actions: cooperate and defect. In real-world prisoner's dilemmas, these choices are temporally extended and different strategies may correspond to sequences of actions, reflecting grades of cooperation. We introduce a Sequential Prisoner's Dilemma (SPD) game to better capture the aforementioned characteristics. In this work, we propose a deep multiagent reinforcement learning approach that investigates the evolution of mutual cooperation in SPD games. Our approach consists of two phases. The first phase is offline: it synthesizes policies with different cooperation degrees and then trains a cooperation degree detection network. The second phase is online: an agent adaptively selects its policy based on the detected degree of opponent cooperation. The effectiveness of our approach is demonstrated in two representative SPD 2D games: the Apple-Pear game and the Fruit Gathering game. Experimental results show that our strategy can avoid being exploited by exploitative opponents and achieve cooperation with cooperative opponents.

Motivation & Objective

• To model real-world social dilemmas where cooperation is not atomic but temporally extended and graded, going beyond classic iterated PD games.
• To develop a scalable, adaptive strategy for multiagent reinforcement learning in large-state-space environments where direct observation of opponent actions is limited.
• To enable agents to detect the cooperation degree of opponents from sequential actions and select appropriate policies to promote mutual cooperation.
• To ensure robustness against exploitative opponents by dynamically shifting to defection when necessary, avoiding long-term losses.

Proposed method

• Proposes a Sequential Prisoner’s Dilemma (SPD) framework to model real-world social dilemmas with temporally extended, graded cooperation behaviors.
• In the offline phase, trains baseline policies with varying cooperation degrees using weighted target rewards and two schemes: IAC and JAC.
• Synthesizes a continuous range of cooperation-level policies from the baseline policies using a policy generation approach.
• Trains an LSTM-based cooperation degree detection network with an encoder-decoder structure to infer opponent cooperation levels from sequential action histories.
• In the online phase, uses the detected cooperation degree to select an appropriate policy from the synthesized policy set, adapting to opponent behavior.
• Extends the Tit-for-Tat principle to SPD by enabling adaptive policy switching based on real-time cooperation degree estimation.

Experimental results

Research questions

• RQ1 How can mutual cooperation be achieved in large-scale, temporally extended social dilemmas where cooperation is not binary but graded?
• RQ2 Can a deep multiagent reinforcement learning agent detect the cooperation level of an opponent from sequential actions without direct observation of their policy?
• RQ3 How can an agent balance cooperation with robustness against exploitation in dynamic environments with switching opponents?
• RQ4 Can adaptive policy selection based on detected cooperation levels outperform fixed strategies like full cooperation or full defection?

Key findings

• In self-play scenarios, agents converged to full mutual cooperation across all initial policy conditions, including the most challenging case where both started with defection policies.
• In the Apple-Pear game, convergence to full cooperation occurred efficiently within a few episodes due to strong visual and behavioral cues from fruit collection.
• In the Fruit Gathering game, convergence took longer due to the reliance on beam emission frequency as a cooperation signal, requiring more observation time.
• Against opponents with changing strategies, the agent achieved higher average rewards than both pure cooperation and pure defection strategies, demonstrating effective adaptation.
• Detection accuracy decreased when opponents switched policies rapidly, but performance remained superior to fixed strategies, especially over longer interaction periods.
• The social welfare (sum of rewards) under the proposed method exceeded that of both pure cooperation and pure defection strategies in all tested scenarios.