[Paper Review] Reflective Oracles: A Foundation for Classical Game Theory

Benja Fallenstein, Jessica Taylor+1 more|arXiv (Cornell University)|Aug 17, 2015

Computability, Logic, AI Algorithms2 references3 citations

TL;DR

This paper introduces reflective oracles—probabilistic oracles that can answer queries about the behavior of oracle machines accessing the same oracle—enabling Turing machines to perform perfect Bayesian reasoning in environments where other agents are embedded as non-distinguished components. The key result is that agents using such oracles almost surely play Nash equilibria, with mixed strategy probabilities arising from the oracle’s inherent randomness, thus providing a foundation for classical game theory without treating players as fundamentally special.

ABSTRACT

Classical game theory treats players as special---a description of a game contains a full, explicit enumeration of all players---even though in the real world, "players" are no more fundamentally special than rocks or clouds. It isn't trivial to find a decision-theoretic foundation for game theory in which an agent's coplayers are a non-distinguished part of the agent's environment. Attempts to model both players and the environment as Turing machines, for example, fail for standard diagonalization reasons. In this paper, we introduce a "reflective" type of oracle, which is able to answer questions about the outputs of oracle machines with access to the same oracle. These oracles avoid diagonalization by answering some queries randomly. We show that machines with access to a reflective oracle can be used to define rational agents using causal decision theory. These agents model their environment as a probabilistic oracle machine, which may contain other agents as a non-distinguished part. We show that if such agents interact, they will play a Nash equilibrium, with the randomization in mixed strategies coming from the randomization in the oracle's answers. This can be seen as providing a foundation for classical game theory in which players aren't special.

Motivation & Objective

• To resolve the foundational issue in classical game theory where players are treated as fundamentally distinct from the environment, despite being physical systems like any other.
• To address the diagonalization problem that arises when modeling agents as Turing machines that reason about each other’s deterministic behavior.
• To develop a decision-theoretic foundation for game theory in which agents are embedded in their environment and can reason about other agents as part of that environment.
• To show that agents using reflective oracles implement causal decision theory and converge to Nash equilibria in multiplayer games.
• To provide a formalism that supports unbounded rationality in embedded agents, clarifying the role of bounded rationality in real-world systems.

Proposed method

• Introduce a reflective oracle that answers queries about the output of oracle machines with access to the same oracle, using probabilistic responses to avoid diagonalization.
• Define a reflection principle where the oracle’s response probabilities align with the actual probabilities of machine outputs, ensuring consistency.
• Model agents as Turing machines with access to a reflective oracle, using causal decision theory to select actions based on expected utility.
• Construct a formal framework where agents treat opponents as part of a probabilistic environment, enabling self-referential reasoning without inconsistency.
• Prove the existence of reflective oracles satisfying a strengthened reflection principle that handles non-halting machines.
• Demonstrate that when multiple agents use reflective oracles, their behavior converges to a Nash equilibrium due to the oracle’s randomization.

Experimental results

Research questions

• RQ1 Can a decision-theoretic foundation for game theory be constructed in which agents are not distinguished from the rest of the environment?
• RQ2 How can agents reason about each other’s behavior when they are computable processes in a shared environment, avoiding diagonalization paradoxes?
• RQ3 Can reflective oracles enable agents to implement causal decision theory in a way that leads to stable game-theoretic outcomes?
• RQ4 What is the relationship between reflective oracles and Nash equilibria in multiplayer games?
• RQ5 Can the framework be extended to handle non-halting computations and uncomputable priors like Solomonoff induction?

Key findings

• Agents using reflective oracles implement causal decision theory and make decisions based on expected utility in a probabilistic environment.
• The randomization in the oracle’s responses leads to mixed strategies in agents’ behavior, with probabilities matching those in a Nash equilibrium.
• When multiple agents interact using reflective oracles, their joint behavior forms a Nash equilibrium, with every Nash equilibrium realizable by some oracle configuration.
• Reflective oracles exist and satisfy a strengthened reflection principle that applies even to non-halting machines, ensuring consistency in probabilistic reasoning.
• The framework supports modeling of predictors that reason about environments they are embedded within, enabling self-referential reasoning in uncomputable settings.
• A reflective oracle version of Solomonoff induction is possible in principle, allowing prediction over hypotheses that include oracle machines.