[Paper Review] The Quest for a Common Model of the Intelligent Decision Maker

Richard S. Sutton|arXiv (Cornell University)|Feb 26, 2022

Cognitive Science and Mapping4 citations

TL;DR

This paper proposes a unified, cross-disciplinary model of the intelligent decision maker—centered on an agent interacting with a world through actions, observations, and rewards—featuring four core components: perception, decision-making, internal evaluation (value function), and a transition model (world model). The key contribution is a shared conceptual framework that standardizes terminology and structure across psychology, neuroscience, AI, economics, and control theory, enabling deeper interdisciplinary collaboration.

ABSTRACT

The premise of the Multi-disciplinary Conference on Reinforcement Learning and Decision Making is that multiple disciplines share an interest in goal-directed decision making over time. The idea of this paper is to sharpen and deepen this premise by proposing a perspective on the decision maker that is substantive and widely held across psychology, artificial intelligence, economics, control theory, and neuroscience, which I call the "common model of the intelligent agent". The common model does not include anything specific to any organism, world, or application domain. The common model does include aspects of the decision maker's interaction with its world (there must be input and output, and a goal) and internal components of the decision maker (for perception, decision-making, internal evaluation, and a world model). I identify these aspects and components, note that they are given different names in different disciplines but refer essentially to the same ideas, and discuss the challenges and benefits of devising a neutral terminology that can be used across disciplines. It is time to recognize and build on the convergence of multiple diverse disciplines on a substantive common model of the intelligent agent.

Motivation & Objective

• To identify and formalize a shared conceptual framework for intelligent decision-making that transcends individual disciplines.
• To address the fragmentation in terminology and conceptualization across psychology, neuroscience, AI, economics, and control theory.
• To establish a neutral, widely applicable model of the intelligent agent that highlights common structural and functional components.
• To facilitate interdisciplinary research by providing a shared vocabulary and reference point for agent design.
• To promote deeper integration of insights from diverse fields by identifying a core model that underlies many existing theories and systems.

Proposed method

• Proposes a general agent-world interaction framework with five key signals: action, observation, reward, state, and goal.
• Identifies four internal components of the agent: perception (mapping observations to internal states), decision-making (policy), internal evaluation (value function), and transition modeling (world model).
• Argues that these components are independently developed across disciplines but fundamentally equivalent in function.
• Introduces neutral terminology—'agent', 'world', 'action', 'observation', 'reward'—to avoid disciplinary bias and promote cross-disciplinary thinking.
• Uses historical and contemporary examples from psychology (Tolman's cognitive map), neuroscience (dopamine as reward-prediction error), control theory, and reinforcement learning to validate the model's broad applicability.
• Positions the model as a foundational reference point for new agent designs, emphasizing how innovations can be understood as extensions or modifications of this common structure.

Experimental results

Research questions

• RQ1 What core structural and functional components are shared across diverse disciplines studying intelligent decision-making?
• RQ2 How can a neutral, interdisciplinary terminology be developed to unify conceptual frameworks in decision science?
• RQ3 To what extent do existing theories in psychology, neuroscience, AI, and control theory converge on a common model of the intelligent agent?
• RQ4 What are the limitations of the proposed common model, and why should it exclude features like intrinsic motivation or auxiliary subtasks?
• RQ5 How can this common model serve as a standard reference for comparing and extending novel agent architectures?

Key findings

• A widely shared, cross-disciplinary model of the intelligent agent exists in practice, though not yet formally recognized or standardized.
• The four components—perception, policy, value function, and transition model—are independently developed and used across psychology, neuroscience, AI, and control theory, indicating deep convergence.
• The reward-prediction error signal in neuroscience (dopamine) corresponds directly to the value function error in reinforcement learning, demonstrating functional equivalence across fields.
• The transition model, or 'world model', is central to theories in cognitive psychology (Tolman's cognitive map), neuroscience (hippocampal function), and modern deep learning (Bengio, LeCun, Schmidhuber).
• The common model provides a stable, minimal framework for comparing agent designs, with the potential to accelerate interdisciplinary progress.
• Despite its simplicity, the model is not exhaustive; features like intrinsic motivation or auxiliary subtasks are excluded to preserve universality and avoid disciplinary bias.