[Paper Review] Agents and Devices: A Relative Definition of Agency

Laurent Orseau, Simon McGregor McGill+1 more|arXiv (Cornell University)|May 31, 2018

Reinforcement Learning in Robotics14 references8 citations

TL;DR

This paper proposes a Bayesian framework to formally assess whether a system should be classified as an agent (goal-optimizing) or a device (input-output mapping) based on its observed behavior. Using inverse reinforcement learning for agents and sequence prediction for devices, it computes posterior probabilities via Bayes' rule, demonstrating that goal-switching behaviors and suboptimal trajectories can still be interpreted as agent-like when modeled with appropriate priors.

ABSTRACT

According to Dennett, the same system may be described using a `physical' (mechanical) explanatory stance, or using an `intentional' (belief- and goal-based) explanatory stance. Humans tend to find the physical stance more helpful for certain systems, such as planets orbiting a star, and the intentional stance for others, such as living animals. We define a formal counterpart of physical and intentional stances within computational theory: a description of a system as either a device, or an agent, with the key difference being that `devices' are directly described in terms of an input-output mapping, while `agents' are described in terms of the function they optimise. Bayes' rule can then be applied to calculate the subjective probability of a system being a device or an agent, based only on its behaviour. We illustrate this using the trajectories of an object in a toy grid-world domain.

Motivation & Objective

• To formalize the observer-relative distinction between agents and devices in computational systems.
• To develop a method for computing the subjective probability that a system is an agent or a device based on its observed behavior.
• To validate the approach on a grid-world domain with diverse behavioral trajectories, including goal-switching and reactive behaviors.
• To demonstrate that agency can be preserved even when goals change, provided the switching is rare and modeled with appropriate priors.
• To provide a non-anthropocentric, formal definition of agency suitable for AI and reinforcement learning research.

Proposed method

• Model the system's behavior as a sequence of input-output pairs (x_t, y_t) over time.
• Define a device mixture M_d as a weighted sum of input-output functions d, each with prior weight w_d, to model mechanical behavior.
• Define an agent mixture M_a using inverse reinforcement learning to infer the most likely goal and ε-greedy policy from observed behavior.
• Apply Bayes' theorem to compute the posterior probability P(M_a | yx_{1:T}) that the system is an agent, given the observed trajectory.
• Compare the log-likelihoods of the trajectory under both device and agent models to determine which explanation is more probable.
• Use a switching goal prior model to allow agents to change goals during behavior, improving fit for complex trajectories.

Experimental results

Research questions

• RQ1 How can we formally quantify the degree to which a system's behavior is best explained as agent-like (goal-optimizing) versus device-like (mechanical input-output mapping)?
• RQ2 To what extent can a system exhibit goal-switching behavior while still being classified as an agent, and how should such behavior be modeled?
• RQ3 How do different behavioral patterns—such as wall-following, random movement, or suboptimal goal pursuit—compare in terms of their likelihood under device versus agent models?
• RQ4 Can the framework distinguish between truly agent-like behavior and reactive, non-goal-directed behavior using only observed trajectories?
• RQ5 How does the observer's prior knowledge influence the posterior probability of agency, and can this be formalized in a computationally tractable way?

Key findings

• The framework successfully distinguishes between device-like and agent-like behaviors in a grid-world domain, with trajectories like wall-following being better explained as devices and goal-directed paths as agents.
• Suboptimal but goal-directed behavior, such as taking a non-optimal path to a target, is more likely under the agent model than the device model, especially when using a switching goal prior.
• Random behavior leads to high negative log-likelihood (NLL) under both models, indicating poor fit for either explanation, consistent with the lack of discernible agency or mechanism.
• When a system switches goals (e.g., from magenta to green balloon), the agent model with a switching prior achieves a significantly better fit than a non-switching model, demonstrating the importance of prior structure.
• The posterior probability of agency increases when the behavior is goal-directed and consistent, even if suboptimal, and decreases for reactive or random behaviors.
• The model shows that a system can maintain agenthood through goal switches, as long as such switches are rare and modeled explicitly in the prior.