[Paper Review] Parametric Bounded Löb's Theorem and Robust Cooperation of Bounded Agents

Andrew Critch|arXiv (Cornell University)|Feb 12, 2016

Game Theory and Applications10 references3 citations

TL;DR

This paper introduces a parametric, bounded version of Löb's Theorem applicable to resource-limited agents that reason about each other's source code. It demonstrates that agents like FairBot_k can achieve mutual cooperation in the Prisoner’s Dilemma not through fragile equality checks, but via proof search within bounded length, leading to robust program equilibria that outperform classical Nash equilibria.

ABSTRACT

Löb's theorem and Gödel's theorems make predictions about the behavior of systems capable of self-reference with unbounded computational resources with which to write and evaluate proofs. However, in the real world, systems capable of self-reference will have limited memory and processing speed, so in this paper we introduce an effective version of Löb's theorem which is applicable given such bounded resources. These results have powerful implications for the game theory of bounded agents who are able to write proofs about themselves and one another, including the capacity to out-perform classical Nash equilibria and correlated equilibria, attaining mutually cooperative program equilibrium in the Prisoner's Dilemma. Previous cooperative program equilibria studied by Tennenholtz (2004) and Fortnow (2009) have depended on tests for program equality, a fragile condition, whereas "Löbian" cooperation is much more robust and agnostic of the opponent's implementation.

Motivation & Objective

• To develop a bounded, effective version of Löb’s Theorem for agents with limited computational resources.
• To enable robust cooperation in game-theoretic scenarios between agents that can inspect each other’s source code.
• To overcome the fragility of prior cooperative program equilibria that depend on exact program equality.
• To demonstrate that mutual cooperation can emerge even in one-shot games when agents use bounded proof search.
• To challenge classical game-theoretic intuitions by showing that cooperation can be logically coherent and strategically optimal under transparency and bounded reasoning.

Proposed method

• Introduces FairBot_k, an agent that searches for a proof of length ≤k that its opponent will cooperate.
• Uses a parametric bound k on proof length to model computational resource limits.
• Applies a bounded version of Löb’s Theorem to show that if a proof exists within k symbols, the agent will cooperate.
• Analyzes mutual cooperation between FairBot_k agents by showing that a proof of mutual cooperation can be derived when k is sufficiently large.
• Demonstrates that the outcome (C,C) is logically coherent and stable under bounded reasoning, even though no agent acts first.
• Reframes decision-making in transparent-agent settings, challenging causal decision theory by showing that logical dependence supersedes causal independence.

Experimental results

Research questions

• RQ1 Can a bounded version of Löb’s Theorem be formulated to apply to agents with finite memory and processing speed?
• RQ2 Under what conditions can two bounded agents achieve mutual cooperation in the Prisoner’s Dilemma via proof search?
• RQ3 How does bounded proof search lead to stable cooperative equilibria that are robust to minor code changes?
• RQ4 Can logical dependence between agents’ decisions override causal independence in strategic decision-making?
• RQ5 What are the implications of such cooperation for the design of advanced AI systems capable of self-reference and mutual reasoning?

Key findings

• FairBot_k achieves mutual cooperation with itself when k is large enough to contain a proof of mutual cooperation, despite no agent initiating cooperation first.
• The outcome (C,C) is logically coherent and stable under bounded reasoning, as shown by the parametric Löb's Theorem.
• Cooperation emerges not from program equality checks but from shared logical reasoning about mutual behavior, making it robust to minor implementation differences.
• The result implies that agents can achieve cooperative program equilibria that outperform classical Nash and correlated equilibria in one-shot games.
• Proof search acts as a logical mechanism that enables cooperation even when causal influence is absent, challenging assumptions of causal decision theory.
• The bounded Löb's Theorem provides a formal foundation for designing cooperative agents in transparent, self-referential systems with limited resources.