[Paper Review] Giving Up Control: Neurons as Reinforcement Learning Agents

Jordan Ott|arXiv (Cornell University)|Mar 17, 2020

Reinforcement Learning in Robotics30 references4 citations

TL;DR

This paper proposes modeling individual neurons as independent reinforcement learning agents that maximize their own self-interest while competing and cooperating within a network. By introducing biologically inspired rewards—such as task performance, sparsity, and activity traces—neural networks achieve stable, scalable learning on high-level tasks like CartPole, outperforming pure task-reward training, especially in deeper architectures.

ABSTRACT

Artificial Intelligence has historically relied on planning, heuristics, and handcrafted approaches designed by experts. All the while claiming to pursue the creation of Intelligence. This approach fails to acknowledge that intelligence emerges from the dynamics within a complex system. Neurons in the brain are governed by local rules, where no single neuron, or group of neurons, coordinates or controls the others. This local structure gives rise to the appropriate dynamics in which intelligence can emerge. Populations of neurons must compete with their neighbors for resources, inhibition, and activity representation. At the same time, they must cooperate, so the population and organism can perform high-level functions. To this end, we introduce modeling neurons as reinforcement learning agents. Where each neuron may be viewed as an independent actor, trying to maximize its own self-interest. By framing learning in this way, we open the door to an entirely new approach to building intelligent systems.

Motivation & Objective

• To overcome the limitations of top-down, behavior-level cost function optimization in AI by shifting focus to bottom-up, emergent intelligence through local neuronal dynamics.
• To address the scalability and stability issues in deep reinforcement learning by embedding biologically plausible reward mechanisms at the neuronal level.
• To demonstrate that neurons acting as independent RL agents can collectively solve high-level tasks through intrinsic competition and cooperation.
• To explore whether local, neuron-specific rewards—such as sparsity and activity traces—can improve network dynamics and enable training of deeper networks.
• To move beyond externally defined reward functions by investigating internal, agent-localized reward generation for more biologically plausible learning.

Proposed method

• Each neuron is modeled as an independent reinforcement learning agent, acting to maximize its own self-interest using policy gradient methods.
• The network uses a multi-agent framework where neurons compete for activity representation and resources while cooperating to achieve high-level task goals.
• Reward signals are composed of three components: task-level reward (e.g., CartPole score), sparsity reward (penalizing over- or under-activity), and activity trace (encouraging balanced firing patterns).
• The activity trace reward penalizes neurons that are consistently active or inactive, promoting diversity in population coding and preventing saturation.
• Training is conducted using the OpenAI Gym CartPole environment with networks of 1, 2, and 5 layers, using stochastic policy optimization.
• Reward schemes are tested in combination: task-only, biologically motivated rewards only, and a hybrid of both to assess impact on learning efficiency and scalability.

Experimental results

Research questions

• RQ1 Can a network of neurons modeled as independent reinforcement learning agents learn to solve a high-level control task like CartPole without centralized supervision?
• RQ2 Does the inclusion of biologically inspired rewards—such as sparsity and activity trace—improve training stability and performance in deeper networks?
• RQ3 How does the performance of a multi-agent neuronal network compare when relying solely on task-level rewards versus incorporating local, biologically grounded rewards?
• RQ4 Can neurons learning for self-interest still achieve collective cooperation necessary for high-level task performance?
• RQ5 To what extent do internal, neuron-localized reward mechanisms reduce reliance on externally defined reward functions?

Key findings

• Networks trained with only the task-level reward failed to scale effectively beyond two layers, requiring over 20,000 episodes to reach the target performance in deeper architectures.
• The inclusion of biologically motivated rewards—sparsity and activity trace—enabled successful training of five-layer networks, reducing the average training time to 4,419 episodes when combined with task rewards.
• The hybrid reward scheme (biological + task) achieved the best performance, with a mean of 4,419 episodes required to reach a 300 average reward over 100 episodes in the five-layer network.
• The activity trace reward successfully enforced balanced firing patterns, preventing neurons from becoming permanently active or silent, thus promoting population-level coding diversity.
• The results demonstrate that biologically inspired rewards are essential for stabilizing learning dynamics in deeper, more complex networks.
• The framework proves feasible for multi-agent, neuron-level RL, showing that self-interested agents can achieve collective intelligence through intrinsic reward structures.