[Paper Review] Discovering Agents

Zachary Kenton, Ramana Kumar+4 more|arXiv (Cornell University)|Aug 17, 2022

Bayesian Modeling and Causal Inference7 citations

TL;DR

This paper proposes the first formal causal definition of agents as systems that adapt their behavior when they expect their actions to influence the world differently. It introduces a causal discovery algorithm to identify agents from empirical data, translating between structural causal models and game-theoretic influence diagrams, enabling rigorous safety analysis of goal-directed systems in AI.

ABSTRACT

Causal models of agents have been used to analyse the safety aspects of machine learning systems. But identifying agents is non-trivial -- often the causal model is just assumed by the modeler without much justification -- and modelling failures can lead to mistakes in the safety analysis. This paper proposes the first formal causal definition of agents -- roughly that agents are systems that would adapt their policy if their actions influenced the world in a different way. From this we derive the first causal discovery algorithm for discovering agents from empirical data, and give algorithms for translating between causal models and game-theoretic influence diagrams. We demonstrate our approach by resolving some previous confusions caused by incorrect causal modelling of agents.

Motivation & Objective

• To address the lack of formal criteria for identifying agents in machine learning systems, especially in safety-critical contexts.
• To resolve ambiguities in prior causal modeling of agents that led to flawed safety analyses.
• To develop a method grounded in causal discovery that empirically identifies agents from data, rather than assuming agenthood.
• To formalize the concept of agency via causal mechanisms that respond to changes in environmental influence.
• To enable automated inference of agent incentives and decision rules using experimental interventions.

Proposed method

• Proposes mechanised structural causal models (SCMs) with explicit mechanism nodes representing decision rules and environmental parameters.
• Develops an algorithm to infer the mechanised causal graph from interventional distributions, enabling causal discovery of agents.
• Introduces a translation algorithm between mechanised SCMs and game-theoretic influence diagrams, linking causal and game-theoretic representations.
• Uses intervention-based experiments to test whether a system would change its behavior under altered causal relationships, defining agency via behavioral sensitivity to counterfactual world models.
• Applies the independent causal mechanisms (ICM) principle to identify violations when agent decision rules are involved, limiting ICM's applicability to non-agent systems.
• Employs soft interventions on variables like cheese location and slip probability to simulate changes in causal structure and assess agent responsiveness.

Experimental results

Research questions

• RQ1 How can we formally define an agent in causal terms, such that it captures the intuition that agents adapt their behavior when they expect different causal outcomes?
• RQ2 What experimental procedures can be used to discover agents from empirical data, without assuming agenthood a priori?
• RQ3 How can we translate between causal models and game-theoretic representations to enable safety analysis of goal-directed systems?
• RQ4 In what ways do standard causal discovery methods, such as ICM, fail when applied to systems containing agents?
• RQ5 Can we resolve past confusions in agent modeling by using intervention-based causal discovery to validate or correct prior assumptions?

Key findings

• The paper establishes a formal causal definition of agents as systems whose behavior would change if they expected their actions to influence the world differently, grounded in counterfactual reasoning.
• The proposed causal discovery algorithm successfully identifies agents by detecting behavioral adaptation under interventions on causal mechanisms, such as changing cheese location or slip probability in the mouse example.
• The method resolves prior confusions in agent modeling by distinguishing systems that are accidentally optimal (like a rock blocking a pipe) from those that are genuinely agentic.
• The approach reveals that the independent causal mechanisms (ICM) principle does not apply to agent decision rules or strategically relevant variables, limiting ICM’s scope to non-agent systems.
• The translation algorithms between mechanised SCMs and game graphs allow for systematic safety analysis of AI systems by deriving incentive structures from causal data.
• Empirical validation in simulated environments, such as the gridworld mouse example, confirms that the method correctly identifies agents based on their sensitivity to causal changes.