[Paper Review] Meta-Learning through Hebbian Plasticity in Random Networks

Elias Najarro, Sebastian Risi|arXiv (Cornell University)|Jul 6, 2020

Advanced Memory and Neural Computing58 references22 citations

TL;DR

This paper proposes a meta-learning approach that evolves synapse-specific Hebbian plasticity rules for randomly initialized neural networks, enabling lifelong adaptation without explicit rewards. The method achieves rapid, robust performance in vision-based and 3D locomotion tasks—such as navigating damaged quadruped robots in under 100 timesteps—by allowing weights to self-organize via local, Hebbian learning rules that converge to high-performing attractors in weight space.

ABSTRACT

Lifelong learning and adaptability are two defining aspects of biological agents. Modern reinforcement learning (RL) approaches have shown significant progress in solving complex tasks, however once training is concluded, the found solutions are typically static and incapable of adapting to new information or perturbations. While it is still not completely understood how biological brains learn and adapt so efficiently from experience, it is believed that synaptic plasticity plays a prominent role in this process. Inspired by this biological mechanism, we propose a search method that, instead of optimizing the weight parameters of neural networks directly, only searches for synapse-specific Hebbian learning rules that allow the network to continuously self-organize its weights during the lifetime of the agent. We demonstrate our approach on several reinforcement learning tasks with different sensory modalities and more than 450K trainable plasticity parameters. We find that starting from completely random weights, the discovered Hebbian rules enable an agent to navigate a dynamical 2D-pixel environment; likewise they allow a simulated 3D quadrupedal robot to learn how to walk while adapting to morphological damage not seen during training and in the absence of any explicit reward or error signal in less than 100 timesteps. Code is available at https://github.com/enajx/HebbianMetaLearning.

Motivation & Objective

• To develop a meta-learning framework that enables lifelong adaptation in artificial agents using only Hebbian plasticity rules.
• To investigate whether randomly initialized networks can achieve high performance in complex RL tasks through evolved local learning rules.
• To demonstrate that Hebbian plasticity can enable fast adaptation to unseen morphological damage without explicit reward signals.
• To explore the potential of plasticity rules as the primary learnable component, rather than network weights, in reinforcement learning.
• To inspire new research directions in neuro-inspired learning by challenging the assumption that static, pre-trained weights are necessary for performance.

Proposed method

• The method evolves connection-specific Hebbian learning rules using a neuroevolution approach, optimizing for performance on reinforcement learning tasks.
• Networks are initialized with random weights from a uniform distribution U[-0.1, 0.1], with no pre-training or fine-tuning.
• Plasticity rules are based on local pre- and post-synaptic activity correlations, following the principle: Δw ∝ pre × post.
• The learning rules are applied continuously during agent deployment, allowing dynamic weight updates throughout the agent’s lifetime.
• The approach uses a differentiable evolution strategy to optimize plasticity rules over multiple random initializations and task instances.
• Performance is evaluated in both vision-based control (CarRacing) and 3D quadruped locomotion tasks, including morphological damage not seen during training.

Experimental results

Research questions

• RQ1 Can Hebbian plasticity rules evolved in random networks enable fast, reward-free adaptation to complex RL tasks?
• RQ2 Can such networks outperform fixed-weight networks in generalization and robustness to morphological damage?
• RQ3 Does self-organization via Hebbian rules lead to convergence to stable, high-performing weight attractors in weight space?
• RQ4 How robust is the performance of evolved Hebbian networks when a large portion of weights are zeroed out?
• RQ5 Can local, non-backpropagated plasticity mechanisms achieve performance comparable to gradient-based methods in complex control tasks?

Key findings

• The Hebbian network achieved high performance in the CarRacing environment across all 100 rollouts, demonstrating robustness and consistency.
• In the 3D quadruped task, the Hebbian network successfully adapted to morphological damage not encountered during training, while the fixed-weight network failed.
• Adaptation occurred in fewer than 100 timesteps, even without any explicit reward or error signal during deployment.
• The evolved Hebbian rules caused the network weights to converge to a stable attractor in weight space, enabling rapid performance improvement.
• The Hebbian network remained functional and adaptive even when up to 50% of its weights were randomly zeroed out, indicating high resilience.
• The approach outperformed static-weight networks in both performance and adaptability, particularly in dynamic, unforeseen conditions.