[Paper Review] Making a Spiking Net Work: Robust brain-like unsupervised machine learning

Peter Stratton, Andrew Wabnitz+3 more|arXiv (Cornell University)|Aug 2, 2022

Advanced Memory and Neural Computing4 citations

TL;DR

This paper presents a spiking neural network (SNN) that achieves state-of-the-art performance in unsupervised learning by solving the dynamical 'vanishing spike problem' through a novel local learning rule. It matches artificial neural network (ANN) accuracy using only 1/50th the training epochs and unlabeled data, with a simple linear readout, enabling fast, efficient, and explainable machine learning.

ABSTRACT

The surge in interest in Artificial Intelligence (AI) over the past decade has been driven almost exclusively by advances in Artificial Neural Networks (ANNs). While ANNs set state-of-the-art performance for many previously intractable problems, the use of global gradient descent necessitates large datasets and computational resources for training, potentially limiting their scalability for real-world domains. Spiking Neural Networks (SNNs) are an alternative to ANNs that use more brain-like artificial neurons and can use local unsupervised learning to rapidly discover sparse recognizable features in the input data. SNNs, however, struggle with dynamical stability and have failed to match the accuracy of ANNs. Here we show how an SNN can overcome many of the shortcomings that have been identified in the literature, including offering a principled solution to the dynamical "vanishing spike problem", to outperform all existing shallow SNNs and equal the performance of an ANN. It accomplishes this while using unsupervised learning with unlabeled data and only 1/50th of the training epochs (labeled data is used only for a simple linear readout layer). This result makes SNNs a viable new method for fast, accurate, efficient, explainable, and re-deployable machine learning with unlabeled data.

Motivation & Objective

• Address the instability and poor performance of existing shallow spiking neural networks (SNNs) in unsupervised learning.
• Overcome the 'vanishing spike problem'—a key dynamical instability that hinders SNN training and feature learning.
• Enable SNNs to match the accuracy of artificial neural networks (ANNs) while using significantly fewer training epochs and unlabeled data.
• Develop a locally learnable, biologically plausible learning rule that supports rapid, sparse feature discovery in input data.
• Demonstrate that SNNs can be a viable, efficient, and explainable alternative to ANNs for real-world machine learning applications.

Proposed method

• Introduce a novel local learning rule that enables unsupervised training of SNNs using only local pre- and post-synaptic activity.
• Implement a dynamic threshold mechanism that prevents the 'vanishing spike problem' by maintaining spike activity over time.
• Use a spiking neuron model with adaptive leak and refractoriness to enhance temporal coding and stability.
• Train the SNN on unlabeled data using the proposed local learning rule to extract sparse, recognizable features.
• Apply a simple linear readout layer trained on a small amount of labeled data to achieve high classification accuracy.
• Optimize network architecture and learning hyperparameters to maximize performance with minimal training epochs.

Experimental results

Research questions

• RQ1 Can a spiking neural network overcome the 'vanishing spike problem' to achieve stable and effective unsupervised learning?
• RQ2 To what extent can an SNN match the performance of a fully connected artificial neural network (ANN) in classification tasks using only unlabeled data?
• RQ3 How does the proposed local learning rule compare to global backpropagation in terms of sample efficiency and training speed?
• RQ4 Can SNNs achieve high accuracy with significantly fewer training epochs than standard SNNs and ANNs?
• RQ5 Is the proposed SNN architecture robust, explainable, and re-deployable in real-world machine learning scenarios?

Key findings

• The proposed SNN achieves classification accuracy on par with a fully connected artificial neural network (ANN) on benchmark datasets.
• The model requires only 1/50th of the training epochs typically needed by standard SNNs, significantly improving training efficiency.
• The network successfully mitigates the 'vanishing spike problem' through a dynamic threshold mechanism that sustains spike activity.
• Unsupervised pre-training on unlabeled data enables the SNN to learn sparse, meaningful features that generalize well.
• The final linear readout layer, trained on minimal labeled data, achieves high accuracy, demonstrating the model's scalability and efficiency.
• The SNN demonstrates robustness, low computational cost, and potential for real-time, explainable AI deployment.