[Paper Review] Predictive representations: building blocks of intelligence

Wilka Carvalho, Momchil S. Tomov+3 more|arXiv (Cornell University)|Feb 9, 2024

Cognitive Science and Education Research4 citations

TL;DR

This paper proposes that predictive representations—particularly the successor representation (SR) and its generalizations—serve as fundamental building blocks of intelligence by enabling efficient, flexible, and scalable decision-making in both artificial and biological systems. By caching task-relevant predictions, these representations allow rapid inference under computational constraints, with strong theoretical, AI, cognitive science, and neuroscience support for their role in exploration, transfer learning, hierarchical control, and neural computation.

ABSTRACT

Adaptive behavior often requires predicting future events. The theory of reinforcement learning prescribes what kinds of predictive representations are useful and how to compute them. This paper integrates these theoretical ideas with work on cognition and neuroscience. We pay special attention to the successor representation (SR) and its generalizations, which have been widely applied both as engineering tools and models of brain function. This convergence suggests that particular kinds of predictive representations may function as versatile building blocks of intelligence.

Motivation & Objective

• To identify and formalize predictive representations that serve as efficient, reusable components for intelligent behavior across artificial and biological systems.
• To address the flexibility-efficiency trade-off in intelligent systems by proposing predictive representations that cache answers to common queries, reducing computational cost.
• To unify insights from reinforcement learning, cognitive science, and neuroscience by demonstrating that the brain likely uses similar predictive representations for decision-making, navigation, and memory.
• To show that generalized predictive representations—such as successor features and successor models—enable powerful capabilities like transfer learning, hierarchical planning, and multi-agent coordination in AI.
• To provide biologically plausible learning algorithms for these representations, linking them to neural mechanisms such as dopamine signaling and hippocampal replay.

Proposed method

• Formalizes predictive representations as cached solutions to specific prediction queries, distinguishing them from full predictive models to balance flexibility and efficiency.
• Introduces the successor representation (SR) as a matrix that encodes expected future state visitation under a policy, enabling fast value function inference via linear algebra.
• Extends SR to successor features (SFs), which generalize predictions to arbitrary features of future states, enabling transfer across tasks with different reward functions.
• Proposes successor models (SMs) as probabilistic generalizations of SR, allowing for sampling from future state distributions and enabling richer planning and inference.
• Applies generalized policy improvement (GPI) to combine multiple behaviors via weighted combinations of SFs, enabling adaptive behavior selection.
• Introduces the Option Keyboard framework to chain options using SR, enabling hierarchical reinforcement learning with efficient option discovery and transfer.

Figure 1: Algorithmic solutions to the RL problem . An agent solving a three-armed maze (bottom) can adopt different classes of strategies (top). Model-based strategies (left) learn an internal model of the environment, including the transition function ( $T$ ), the reward function ( $R$ ), and (opt

Experimental results

Research questions

• RQ1 What kinds of predictive representations are most effective for efficient and flexible decision-making in reinforcement learning?
• RQ2 How can predictive representations be generalized beyond the standard SR to support transfer learning, hierarchical control, and multi-task learning?
• RQ3 What neural mechanisms in the brain could implement predictive representations like the SR and SFs, and how do they relate to observed neural activity patterns?
• RQ4 How do dopamine neurons encode vector-valued prediction errors that support learning of successor features and models?
• RQ5 To what extent do predictive representations unify artificial intelligence, cognitive science, and neuroscience in explaining intelligent behavior?

Key findings

• The successor representation (SR) enables fast value function inference by caching expected future state visitation, significantly reducing computational cost compared to full predictive models.
• Successor features (SFs) allow transfer across tasks with different reward functions by decoupling prediction of future states from reward shaping, enabling efficient reweighting of value functions.
• Successor models (SMs) generalize SR to probabilistic predictions of future states, supporting both sampling-based and evaluation-based inference, which enhances planning and exploration.
• Neural data from fMRI studies show that human sequential neural activity during memory recall mirrors predictions from SR models trained on graph-structured environments, supporting a neural basis for SR.
• Dopamine neuron ensembles encode vector-valued prediction errors, including sensory feature predictions, which supports the learning of successor features and explains non-scalar dopamine responses.
• The temporal context model (TCM) in memory systems can be interpreted as a biologically plausible estimator for the SR, linking episodic memory and decision-making through predictive representations.

Figure 2: The successor representation (SR). We present (a) a schematic of an environment where the agent is a red box at state $s_{13}$ and the goal is a green box at state $s_{5}$ . In general, an SR $M^{\pi}(s,\tilde{s})$ (Eq. 13 ) describes the discounted state occupancy for state $\tilde{s}$ wh