[Paper Review] Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic

Mihály Bárász, Paul F. Christiano+4 more|arXiv (Cornell University)|Jan 22, 2014

Game Theory and Applications18 references17 citations

TL;DR

This paper introduces modal agents that achieve robust cooperation in one-shot Prisoner’s Dilemma games by using provability logic to reason about opponents' source code. It constructs agents like PrudentBot that cooperate if they can prove mutual cooperation is logically consistent, ensuring unexploitability and robustness even when source codes are syntactically different, with Löb's Theorem enabling stable fixed-point reasoning in counterfactuals.

ABSTRACT

We consider the one-shot Prisoner's Dilemma between algorithms with read-access to one anothers' source codes, and we use the modal logic of provability to build agents that can achieve mutual cooperation in a manner that is robust, in that cooperation does not require exact equality of the agents' source code, and unexploitable, meaning that such an agent never cooperates when its opponent defects. We construct a general framework for such "modal agents", and study their properties.

Motivation & Objective

• To develop agents that cooperate in one-shot Prisoner’s Dilemma games without requiring exact source code equality.
• To ensure unexploitability, so an agent never cooperates when the opponent defects.
• To design agents that achieve mutual cooperation through logical reasoning about provability rather than syntactic matching.
• To formalize a framework for program equilibrium using modal logic, enabling stable counterfactual reasoning in game-theoretic settings.

Proposed method

• Uses Gödel-Löb modal logic (GL) to formalize agents' reasoning about provability of cooperation.
• Defines agents such as FairBot and PrudentBot based on provability conditions: e.g., cooperate if the system proves the opponent will cooperate.
• Applies Löb’s Theorem to ensure fixed-point consistency: if a system proves that cooperation leads to mutual cooperation, then cooperation actually occurs.
• Constructs modal agents that reason about their opponents’ source code via logical formulas, avoiding infinite regress through provability checks.
• Uses quining and self-reference techniques to embed logical conditions into agent behavior.
• Establishes a general framework for modal agents where actions are determined by provability of logical formulas involving opponent behavior.

Experimental results

Research questions

• RQ1 Can we design agents that cooperate robustly in one-shot Prisoner’s Dilemma without requiring syntactic identity of source code?
• RQ2 How can we ensure that an agent never cooperates when the opponent defects, thus achieving unexploitability?
• RQ3 Can provability logic provide a stable foundation for counterfactual reasoning in program equilibrium?
• RQ4 What logical conditions allow for mutual cooperation even when agents are semantically distinct but logically consistent?
• RQ5 How can modal agents be extended to more complex games beyond the Prisoner’s Dilemma?

Key findings

• PrudentBot achieves mutual cooperation with FairBot and CooperateBot while remaining unexploitable, as it only cooperates if it can prove the opponent will not defect.
• The use of provability logic enables stable fixed points in counterfactual reasoning, avoiding infinite regress in mutual simulation.
• Löb’s Theorem ensures that if a system proves that cooperation leads to mutual cooperation, then cooperation actually occurs, providing logical consistency.
• Modal agents can be constructed to cooperate with a broad class of semantically distinct but logically consistent agents, not just syntactically identical ones.
• The framework supports robust cooperation even when agents have access to each other’s source code, provided logical consistency is maintained.
• The approach demonstrates that unexploitable and robust cooperation is logically possible in one-shot games under provability-based reasoning.