[Paper Review] Complex behavior from intrinsic motivation to occupy action-state path space

Jorge Ramírez‐Ruiz, Dmytro Grytskyy+3 more|arXiv (Cornell University)|May 20, 2022

Evolutionary Game Theory and Cooperation4 citations

TL;DR

This paper proposes that intrinsic motivation to maximize future action-state path entropy—rather than external rewards—generates complex, goal-directed behaviors like dancing, hide-and-seek, and altruism in agents. By formulating a Bellman equation for entropy maximization over action-state trajectories, the authors prove uniqueness and convergence of the optimal policy, demonstrating that agents naturally explore rich, diverse behaviors without reward shaping or external objectives.

ABSTRACT

Most theories of behavior posit that agents tend to maximize some form of reward or utility. However, animals very often move with curiosity and seem to be motivated in a reward-free manner. Here we abandon the idea of reward maximization, and propose that the goal of behavior is maximizing occupancy of future paths of actions and states. According to this maximum occupancy principle, rewards are the means to occupy path space, not the goal per se; goal-directedness simply emerges as rational ways of searching for resources so that movement, understood amply, never ends. We find that action-state path entropy is the only measure consistent with additivity and other intuitive properties of expected future action-state path occupancy. We provide analytical expressions that relate the optimal policy and state-value function, and prove convergence of our value iteration algorithm. Using discrete and continuous state tasks, including a high--dimensional controller, we show that complex behaviors such as `dancing', hide-and-seek and a basic form of altruistic behavior naturally result from the intrinsic motivation to occupy path space. All in all, we present a theory of behavior that generates both variability and goal-directedness in the absence of reward maximization.

Motivation & Objective

• To address the fundamental problem of defining external rewards in reinforcement learning, which is often arbitrary and task-dependent.
• To propose a new intrinsic motivation mechanism where agents maximize long-term occupancy of action-state paths, not external rewards.
• To formalize action-state path entropy as the only measure satisfying additivity, positivity, and derivability for future path occupancy.
• To demonstrate that entropy maximization alone generates complex, adaptive, and seemingly goal-directed behaviors in both discrete and continuous environments.
• To provide a mathematically rigorous framework with unique solution and convergence guarantees for entropy-based policy optimization.

Proposed method

• The agent’s objective is to maximize discounted future action-state path entropy, modeled as a Markov decision process with no external rewards.
• The optimal policy is derived from a Bellman equation that balances action and state entropy with hyperparameters α and β.
• The state-value function is computed via iterative value iteration, with linear interpolation used for continuous state spaces.
• The method uses deterministic transitions for the agent and stochastic transitions for other entities (e.g., pets), enabling sum over successor states in the Bellman update.
• The algorithm is validated in discrete (agent-pet, prey-predator) and continuous (cartpole) environments with varying β to control state vs. action entropy weighting.
• Value function normalization is applied post-interpolation to ensure valid probability policies.

Experimental results

Research questions

• RQ1 Can agents generate complex, seemingly goal-directed behaviors such as dancing and hide-and-seek without any external reward shaping?
• RQ2 Is action-state path entropy the only measure that satisfies additivity and intuitive properties for future path occupancy?
• RQ3 Does maximizing action-state path entropy lead to unique and convergent solutions in the Bellman equation framework?
• RQ4 Can intrinsic exploration through entropy maximization naturally lead to altruistic behaviors, such as opening a gate for a pet?
• RQ5 How does the balance between action entropy and state entropy (controlled by β) affect the emergence of complex behaviors?

Key findings

• Agents maximizing action-state path entropy spontaneously exhibit 'dancing' behavior in the cartpole environment, even without external rewards.
• In a prey-predator task, agents naturally perform hide-and-seek by exploiting environmental constraints and stochastic transitions.
• In the agent-pet scenario, the agent learns to open the gate (increasing gate-open duration) when β=1, demonstrating a basic form of altruism.
• The fraction of time the gate remained open increased from 0.05 (β=0) to 0.85 (β=1), showing a strong dependence on state entropy weighting.
• Heatmaps confirmed that higher β values led to more uniform and extensive occupation of the pet’s reachable states, indicating richer exploration.
• Theoretical analysis proved that the Bellman equation for action-state path entropy has a unique solution and converges under the iterative update rule.