[Paper Review] Hebbian Deep Learning Without Feedback

Adrien Journé, Hector Garcia Rodriguez+2 more|arXiv (Cornell University)|Sep 23, 2022

Advanced Memory and Neural Computing17 citations

TL;DR

This paper introduces SoftHebb, a bio-plausible deep learning algorithm that trains deep neural networks without feedback, error signals, or target supervision by leveraging Hebbian learning in soft winner-take-all networks. It achieves state-of-the-art accuracy on MNIST (99.4%), CIFAR-10 (80.3%), STL-10 (76.2%), and ImageNet (27.3%) using only a single unsupervised pre-training epoch, while eliminating weight transport, non-local plasticity, and update locking—key limitations of backpropagation and prior bio-plausible methods.

ABSTRACT

Recent approximations to backpropagation (BP) have mitigated many of BP's computational inefficiencies and incompatibilities with biology, but important limitations still remain. Moreover, the approximations significantly decrease accuracy in benchmarks, suggesting that an entirely different approach may be more fruitful. Here, grounded on recent theory for Hebbian learning in soft winner-take-all networks, we present multilayer SoftHebb, i.e. an algorithm that trains deep neural networks, without any feedback, target, or error signals. As a result, it achieves efficiency by avoiding weight transport, non-local plasticity, time-locking of layer updates, iterative equilibria, and (self-) supervisory or other feedback signals -- which were necessary in other approaches. Its increased efficiency and biological compatibility do not trade off accuracy compared to state-of-the-art bio-plausible learning, but rather improve it. With up to five hidden layers and an added linear classifier, accuracies on MNIST, CIFAR-10, STL-10, and ImageNet, respectively reach 99.4%, 80.3%, 76.2%, and 27.3%. In conclusion, SoftHebb shows with a radically different approach from BP that Deep Learning over few layers may be plausible in the brain and increases the accuracy of bio-plausible machine learning. Code is available at https://github.com/NeuromorphicComputing/SoftHebb.

Motivation & Objective

• To develop a deep learning algorithm that operates without backpropagation, feedback signals, or error targets, addressing key biological and computational limitations of backpropagation.
• To eliminate the weight transport problem, non-local plasticity, and update-locking mechanisms inherent in backpropagation and other bio-plausible learning methods.
• To achieve high accuracy in deep networks using only unsupervised, Hebbian-style learning with local, synaptic plasticity rules.
• To demonstrate that deep learning without feedback is feasible and accurate in biological and neuromorphic computing contexts.

Proposed method

• Proposes SoftHebb, a multilayer deep learning algorithm based on recent theory of Hebbian learning in soft winner-take-all (WTA) networks.
• Employs a norm-dependent adaptive learning rate that adjusts based on the magnitude of weight vectors, enabling stable and fast convergence to unit norm.
• Uses local, Hebbian-style plasticity rules where synaptic weights are updated based only on pre- and post-synaptic activity, without error signals or global feedback.
• Applies a soft WTA mechanism in each layer to enforce competition among neurons, promoting sparse and distributed representations.
• Employs a single unsupervised pre-training phase with no backpropagation or target supervision, followed by a linear classifier for final prediction.
• Utilizes RePU and softmax activation functions with tuned hyperparameters to optimize representation learning in deep layers.

Experimental results

Research questions

• RQ1 Can deep neural networks be trained effectively without any feedback, error signals, or target supervision?
• RQ2 Is it possible to achieve high accuracy in deep learning while eliminating the weight transport problem and non-local plasticity?
• RQ3 Can Hebbian learning in soft WTA networks support hierarchical feature learning comparable to backpropagation?
• RQ4 How does the proposed norm-dependent adaptive learning rate affect convergence and performance in deep networks?
• RQ5 What is the representational capacity of SoftHebb in learning complex, hierarchical features on standard vision benchmarks?

Key findings

• SoftHebb achieves 99.4% accuracy on MNIST using five hidden layers and a linear classifier, outperforming most bio-plausible methods.
• On CIFAR-10, SoftHebb reaches 80.3% accuracy with only one unsupervised pre-training epoch, surpassing all other feedback-free methods except backpropagation.
• On STL-10, SoftHebb attains 76.2% accuracy, demonstrating strong generalization in deeper, more complex datasets.
• On ImageNet, SoftHebb achieves 27.3% top-1 accuracy, a significant result for a feedback-free, unsupervised method.
• The norm-dependent adaptive learning rate enables faster convergence to unit-norm weight vectors and improves robustness across diverse weight initializations.
• Receptive fields in deeper layers of SoftHebb exhibit increasing complexity, including mixtures of Gabor-like filters and varied textures, indicating hierarchical feature learning.