[Paper Review] Multi-layer Hebbian networks with modern deep learning frameworks

Thomas Miconi|arXiv (Cornell University)|Jul 4, 2021

Neural dynamics and brain function5 references4 citations

TL;DR

This paper demonstrates that hierarchical, convolutional Hebbian networks can be efficiently implemented using modern deep learning frameworks by designing specific loss functions whose gradients exactly replicate biologically plausible learning rules—such as Hebbian, instar, and Oja's rules. The approach enables multi-layer object recognition with improved performance through interventions like sparse plasticity and pruning, which counteract feature degeneracy and allow information to increase across layers.

ABSTRACT

Deep learning networks generally use non-biological learning methods. By contrast, networks based on more biologically plausible learning, such as Hebbian learning, show comparatively poor performance and difficulties of implementation. Here we show that hierarchical, convolutional Hebbian learning can be implemented almost trivially with modern deep learning frameworks, by using specific losses whose gradients produce exactly the desired Hebbian updates. We provide expressions whose gradients exactly implement a plain Hebbian rule (dw ~= xy), Grossberg's instar rule (dw ~= y(x-w)), and Oja's rule (dw ~= y(x-yw)). As an application, we build Hebbian convolutional multi-layer networks for object recognition. We observe that higher layers of such networks tend to learn large, simple features (Gabor-like filters and blobs), explaining the previously reported decrease in decoding performance over successive layers. To combat this tendency, we introduce interventions (denser activations with sparse plasticity, pruning of connections between layers) which result in sparser learned features, massively increase performance, and allow information to increase over successive layers. We hypothesize that more advanced techniques (dynamic stimuli, trace learning, feedback connections, etc.), together with the massive computational boost offered by modern deep learning frameworks, could greatly improve the performance and biological relevance of multi-layer Hebbian networks.

Motivation & Objective

• To bridge the gap between biologically plausible learning rules and modern deep learning by enabling their practical implementation in multi-layer networks.
• To address the poor performance and implementation challenges of Hebbian learning in deep networks through integration with modern deep learning frameworks.
• To investigate whether Hebbian learning in deep architectures can achieve competitive performance in object recognition tasks.
• To identify and mitigate the tendency of higher layers to learn large, simple features that degrade decoding performance.
• To explore how interventions like sparse plasticity and pruning can enhance feature sparsity and information flow across layers.

Proposed method

• Designing custom loss functions whose gradients exactly implement the plain Hebbian rule (dw ~= xy), Grossberg's instar rule (dw ~= y(x-w)), and Oja's rule (dw ~= y(x-yw)).
• Using modern deep learning frameworks to backpropagate gradients from these losses, enabling end-to-end training of multi-layer convolutional Hebbian networks.
• Applying hierarchical, convolutional architectures to learn features in a layer-by-layer fashion, mimicking cortical processing.
• Introducing denser activations with sparse plasticity to promote sparsity in learned features and prevent degeneracy.
• Implementing pruning of inter-layer connections to reduce redundancy and improve information flow across layers.
• Training networks on object recognition tasks while monitoring feature evolution and decoding performance across layers.

Experimental results

Research questions

• RQ1 Can biologically inspired Hebbian learning rules be effectively implemented in deep convolutional networks using modern deep learning frameworks?
• RQ2 Why do higher layers in Hebbian networks tend to learn large, simple features, and how does this affect performance?
• RQ3 Can interventions such as sparse plasticity and pruning improve feature sparsity and performance in Hebbian networks?
• RQ4 Does information content increase across layers in Hebbian networks when appropriate interventions are applied?
• RQ5 To what extent can modern deep learning frameworks enable the development of more biologically plausible deep learning models?

Key findings

• Higher layers in unmodified Hebbian networks learn large, simple features such as Gabor-like filters and blobs, which correlates with decreased decoding performance.
• The introduction of denser activations with sparse plasticity significantly increases network performance by promoting sparser, more informative features.
• Pruning connections between layers further enhances performance and enables information to increase across successive layers.
• The use of gradient-based losses that exactly implement Hebbian rules allows for trivial integration of these biologically plausible learning mechanisms into modern deep learning pipelines.
• The results suggest that with appropriate architectural and training interventions, Hebbian networks can achieve competitive performance while maintaining biological plausibility.
• The framework opens the door for future exploration of advanced techniques like trace learning and feedback connections in biologically inspired deep learning.