[Paper Review] Is Bio-Inspired Learning Better than Backprop? Benchmarking Bio Learning vs. Backprop

Manas Gupta, Sarthak Ketanbhai Modi+3 more|arXiv (Cornell University)|Dec 9, 2022

Neural Networks and Applications4 citations

TL;DR

This paper benchmarks bio-inspired learning algorithms—Hebbian, Feedback Alignment, Direct Feedback Alignment, Predictive Coding, and Difference Target Propagation—against backpropagation (BP) across multiple realistic training constraints. It finds that bio-algorithms significantly outperform BP in low-data and low-budget settings, with Hebbian learning achieving up to 20% higher accuracy and converging 20× faster (in 5 epochs vs. 100 for BP), while also showing robustness to noise and weight sparsity.

ABSTRACT

Bio-inspired learning has been gaining popularity recently given that Backpropagation (BP) is not considered biologically plausible. Many algorithms have been proposed in the literature which are all more biologically plausible than BP. However, apart from overcoming the biological implausibility of BP, a strong motivation for using Bio-inspired algorithms remains lacking. In this study, we undertake a holistic comparison of BP vs. multiple Bio-inspired algorithms to answer the question of whether Bio-learning offers additional benefits over BP. We test Bio-algorithms under different design choices such as access to only partial training data, resource constraints in terms of the number of training epochs, sparsification of the neural network parameters and addition of noise to input samples. Through these experiments, we notably find two key advantages of Bio-algorithms over BP. Firstly, Bio-algorithms perform much better than BP when the entire training dataset is not supplied. Four of the five Bio-algorithms tested outperform BP by upto 5% accuracy when only 20% of the training dataset is available. Secondly, even when the full dataset is available, Bio-algorithms learn much quicker and converge to a stable accuracy in far lesser training epochs than BP. Hebbian learning, specifically, is able to learn in just 5 epochs compared to around 100 epochs required by BP. These insights present practical reasons for utilising Bio-learning beyond just their biological plausibility and also point towards interesting new directions for future work on Bio-learning.

Motivation & Objective

• To evaluate whether bio-inspired learning algorithms offer practical advantages beyond biological plausibility compared to backpropagation.
• To investigate performance differences under realistic constraints such as limited training data, reduced training epochs, input noise, and model sparsity.
• To identify tangible benefits of bio-learning for real-world applications like edge AI and data-scarce environments.
• To analyze the representational learning quality of bio-algorithms versus BP through feature visualization.
• To assess robustness of bio-algorithms under extreme parameter sparsity and noisy inputs.

Proposed method

• Benchmarked five bio-inspired algorithms: Hebbian learning, Feedback Alignment (FA), Direct Feedback Alignment (DFA), Predictive Coding (PC), and Difference Target Propagation (DTP).
• Evaluated performance across four experimental settings: 20% training data, limited epochs (up to 100), input noise (Gaussian and pepper), and 95% weight sparsity.
• Used standard datasets: CIFAR-10, CIFAR-100, and SST-2 for NLP sentiment classification.
• Trained models over 10 random seeds to compute mean accuracy and standard deviation.
• Visualized learned features using activation maps to compare interpretability and structure between BP and Hebbian learning.
• Applied network pruning at 95% sparsity and evaluated robustness to noise using standard image corruption techniques.

Experimental results

Research questions

• RQ1 Do bio-inspired learning algorithms outperform backpropagation in scenarios with limited training data?
• RQ2 Can bio-algorithms achieve faster convergence and stable performance with fewer training epochs compared to BP?
• RQ3 How do bio-algorithms perform under input noise and model sparsity compared to BP?
• RQ4 Do bio-algorithms learn more interpretable or fine-grained features than BP?
• RQ5 Do the advantages of bio-learning persist across different domains, such as NLP?

Key findings

• Four out of five bio-algorithms (FA, DFA, HB, PC) outperformed BP by up to 5% accuracy when only 20% of the training data was available.
• Hebbian learning achieved up to 20% higher accuracy than BP under limited data and limited training budget, with convergence in just 5 epochs compared to BP’s ~100 epochs.
• Hebbian learning converged 20× faster than BP, reaching stable accuracy in 5 epochs on multiple datasets.
• Feature visualization revealed that Hebbian learning learned more fine-grained and explainable features compared to the more random-looking features learned by BP.
• Bio-algorithms, especially DFA, outperformed BP under both Gaussian and pepper noise on CIFAR-10 and CIFAR-100, with DFA achieving higher accuracy than BP even under noise.
• Bio-learning maintained high accuracy under extreme sparsity (95% pruned weights), matching or exceeding BP performance, with all algorithms except Hebbian showing robustness to sparsity.