[Paper Review] Technical report: supervised training of convolutional spiking neural networks with PyTorch

Romain Zimmer, Thomas Pellegrini+2 more|arXiv (Cornell University)|Nov 22, 2019

Advanced Memory and Neural ComputingEngineering17 citations

TL;DR

This paper presents a supervised training framework for convolutional spiking neural networks (SNNs) using backpropagation through time with surrogate gradients in PyTorch. It introduces spatial and temporal convolutions, lateral horizontal connections inspired by Denève, and achieves 94% accuracy on the Google Speech Commands dataset with low firing rates (~5Hz), demonstrating state-of-the-art performance for SNNs in event-based learning.

ABSTRACT

Recently, it has been shown that spiking neural networks (SNNs) can be trained efficiently, in a supervised manner, using backpropagation through time. Indeed, the most commonly used spiking neuron model, the leaky integrate-and-fire neuron, obeys a differential equation which can be approximated using discrete time steps, leading to a recurrent relation for the potential. The firing threshold causes optimization issues, but they can be overcome using a surrogate gradient. Here, we extend previous approaches in two ways. Firstly, we show that the approach can be used to train convolutional layers. Convolutions can be done in space, time (which simulates conduction delays), or both. Secondly, we include fast horizontal connections à la Denève: when a neuron N fires, we subtract to the potentials of all the neurons with the same receptive the dot product between their weight vectors and the one of neuron N. As Denève et al. showed, this is useful to represent a dynamic multidimensional analog signal in a population of spiking neurons. Here we demonstrate that, in addition, such connections also allow implementing a multidimensional send-on-delta coding scheme. We validate our approach on one speech classification benchmarks: the Google speech command dataset. We managed to reach nearly state-of-the-art accuracy (94%) while maintaining low firing rates (about 5Hz). Our code is based on PyTorch and is available in open source at http://github.com/romainzimmer/s2net

Motivation & Objective

• To develop a scalable, differentiable framework for training deep convolutional spiking neural networks (SNNs) with backpropagation through time.
• To extend SNN training to include spatial and temporal convolutions, enabling processing of dynamic, time-varying signals like audio.
• To integrate lateral horizontal connections à la Denève to implement multidimensional send-on-delta coding and improve signal representation.
• To achieve high accuracy with sparse, biologically plausible spike activity in SNNs, approaching standard DNN performance.
• To provide an open-source, PyTorch-based implementation to lower the barrier to entry for SNN research.

Proposed method

• Uses the Leaky Integrate-and-Fire (LIF) neuron model with a differential equation approximated via discrete time steps using β = e^(-h/τ).
• Applies surrogate gradient backpropagation to handle the non-differentiable firing threshold, with a gradient scale of 10.
• Implements convolutional layers that operate in both spatial and temporal dimensions, with kernels of size H×W along time and frequency axes.
• Introduces lateral horizontal connections: when a neuron fires, the potentials of neurons in the same receptive field are updated via dot product with its weight vector.
• Employs a fully connected readout layer with L2 regularization on spike activity to enforce sparsity and reduce firing rates.
• Uses Rectified-Adam optimizer with learning rate 10⁻³, weight decay 10⁻⁵, and gradient clipping to stabilize training.

Experimental results

Research questions

• RQ1 Can convolutional spiking neural networks be effectively trained with backpropagation through time using surrogate gradients?
• RQ2 Does the inclusion of lateral horizontal connections improve signal representation and classification performance in SNNs?
• RQ3 Can SNNs achieve high accuracy on speech recognition tasks while maintaining low spike rates?
• RQ4 How does the combination of spatial and temporal convolutions affect performance in event-based learning?
• RQ5 Is recurrence through time necessary when using large time-step discretizations?

Key findings

• The model achieves 94% accuracy on the Google Speech Commands dataset, approaching the performance of standard DNNs (96–97%).
• The network maintains a very low mean firing rate of approximately 5Hz, indicating highly sparse and energy-efficient activity.
• The regularization coefficient effectively controls the trade-off between sparsity and accuracy, as shown in Figure 6.2.
• Lateral horizontal connections enable a multidimensional send-on-delta coding scheme, improving signal representation in the network.
• The model performs comparably without lateral connections, suggesting they are beneficial but not strictly necessary for this task.
• Recurrence through time is not essential when using large time-step discretizations, as independent processing per time step suffices.