[Paper Review] Robust and accelerated single-spike spiking neural network training with applicability to challenging temporal tasks

Luke Taylor, Andrew P. King+1 more|arXiv (Cornell University)|May 30, 2022

Advanced Memory and Neural Computing4 citations

TL;DR

This paper proposes a fast, robust, and differentiable training method for single-spike spiking neural networks (SNNs) that replaces sequential time-dependent operations with parallelizable GPU-accelerated convolutions, achieving up to 13.98× faster training and 81% fewer spikes than multi-spike SNNs, while matching multi-spike SNN performance on challenging temporal neuromorphic datasets through trainable membrane time constants.

ABSTRACT

Spiking neural networks (SNNs), particularly the single-spike variant in which neurons spike at most once, are considerably more energy efficient than standard artificial neural networks (ANNs). However, single-spike SSNs are difficult to train due to their dynamic and non-differentiable nature, where current solutions are either slow or suffer from training instabilities. These networks have also been critiqued for their limited computational applicability such as being unsuitable for time-series datasets. We propose a new model for training single-spike SNNs which mitigates the aforementioned training issues and obtains competitive results across various image and neuromorphic datasets, with up to a $13.98 imes$ training speedup and up to an $81\%$ reduction in spikes compared to the multi-spike SNN. Notably, our model performs on par with multi-spike SNNs in challenging tasks involving neuromorphic time-series datasets, demonstrating a broader computational role for single-spike SNNs than previously believed.

Motivation & Objective

• Address the critical challenge of slow and unstable training in single-spike spiking neural networks (SNNs), which are otherwise highly energy-efficient.
• Overcome the dead neuron problem and eliminate reliance on careful regularization in single-spike SNN training.
• Enable single-spike SNNs to perform competitively on complex temporal tasks, such as neuromorphic time-series datasets, previously considered unsuitable for such networks.
• Develop a generalizable, differentiable training framework that supports diverse architectures and trainable parameters beyond synaptic weights.

Proposed method

• Replace sequential time-dependent operations with parallelizable convolutional operations to accelerate training by eliminating temporal dependencies.
• Use surrogate gradients that propagate through membrane potentials rather than spike times, enabling stable backpropagation in non-differentiable SNNs.
• Introduce trainable membrane time constants to allow neurons to integrate inputs over multiple time scales, enhancing temporal processing capability.
• Design a fully differentiable training pipeline that supports feedforward, hierarchical, and convolutional network architectures.
• Apply batch processing and GPU-optimized operations to maximize computational efficiency while maintaining training stability.
• Ensure robustness to initial zero-activity conditions by avoiding reliance on spike-based gradient signals.

Experimental results

Research questions

• RQ1 Can single-spike SNNs be trained efficiently and stably without sequential time dependence?
• RQ2 Can single-spike SNNs achieve performance comparable to multi-spike SNNs on complex temporal datasets like SHD?
• RQ3 Does introducing trainable membrane time constants enable single-spike SNNs to handle dynamic, time-varying inputs effectively?
• RQ4 Can the training process be accelerated significantly using GPU-parallelizable operations without sacrificing accuracy?
• RQ5 Is the proposed method robust to the dead neuron problem and insensitive to hyperparameter tuning?

Key findings

• The proposed method achieves up to 16.77× training speedup on synthetic benchmarks and up to 13.98× speedup on real-world datasets by replacing sequential operations with parallelizable convolutions.
• The model reduces spike counts by up to 81% compared to multi-spike SNNs while maintaining competitive accuracy across all tested image and neuromorphic datasets.
• The network remains robust to the dead neuron problem and does not require careful regularization, unlike prior single-spike training methods.
• The inclusion of trainable membrane time constants enables single-spike SNNs to perform on par with multi-spike SNNs on challenging temporal datasets such as the SHD neuromorphic dataset.
• The method generalizes across architectures, including deep, feedforward, hierarchical, and convolutional networks, demonstrating broad applicability.
• Experiments confirm that networks with zero initial activity still learn effectively, validating the method’s stability and robustness.