[Paper Review] Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings

Ilyass Hammouamri, Ismail Khalfaoui-Hassani+1 more|arXiv (Cornell University)|Jun 30, 2023

Advanced Memory and Neural ComputingEngineering52 references18 citations

TL;DR

The paper presents a discrete-time backpropagation method to learn synaptic delays in deep feedforward SNNs by modeling delays as 1D temporal convolutions with learnable spacings using DCLS, achieving state-of-the-art results on temporal benchmarks SHD, SSC, and GSC-35 with fewer parameters.

ABSTRACT

Spiking Neural Networks (SNNs) are a promising research direction for building power-efficient information processing systems, especially for temporal tasks such as speech recognition. In SNNs, delays refer to the time needed for one spike to travel from one neuron to another. These delays matter because they influence the spike arrival times, and it is well-known that spiking neurons respond more strongly to coincident input spikes. More formally, it has been shown theoretically that plastic delays greatly increase the expressivity in SNNs. Yet, efficient algorithms to learn these delays have been lacking. Here, we propose a new discrete-time algorithm that addresses this issue in deep feedforward SNNs using backpropagation, in an offline manner. To simulate delays between consecutive layers, we use 1D convolutions across time. The kernels contain only a few non-zero weights - one per synapse - whose positions correspond to the delays. These positions are learned together with the weights using the recently proposed Dilated Convolution with Learnable Spacings (DCLS). We evaluated our method on three datasets: the Spiking Heidelberg Dataset (SHD), the Spiking Speech Commands (SSC) and its non-spiking version Google Speech Commands v0.02 (GSC) benchmarks, which require detecting temporal patterns. We used feedforward SNNs with two or three hidden fully connected layers, and vanilla leaky integrate-and-fire neurons. We showed that fixed random delays help and that learning them helps even more. Furthermore, our method outperformed the state-of-the-art in the three datasets without using recurrent connections and with substantially fewer parameters. Our work demonstrates the potential of delay learning in developing accurate and precise models for temporal data processing. Our code is based on PyTorch / SpikingJelly and available at: https://github.com/Thvnvtos/SNN-delays

Motivation & Objective

• Motivate and exploit delay learning in spiking neural networks to enhance temporal pattern processing.
• Propose a differentiable method to learn synaptic delays jointly with weights in deep SNNs.
• Show equivalence between 1D temporal convolutions and connection delays and use DCLS to learn delays.
• Demonstrate superior accuracy with fewer parameters on SHD, SSC, and GSC-35 without recurrent connections.

Proposed method

• Formulate each synaptic delay as a 1D temporal convolution with a kernel containing a single non-zero element per synapse.
• Use a Gaussian-interpolated DCLS kernel to learn delay positions while gradually shrinking the Gaussian width (sigma) during training.
• Represent delays with learnable positions d_ij and shared sigma, enabling backpropagation through the delay parameters.
• Convert the learned continuous kernels to discrete delays for inference, enabling sparse, hardware-friendly connections.
• Train using surrogate-gradient backpropagation with Leaky Integrate-and-Fire neurons and an offline, feedforward architecture.
• Compare with state-of-the-art methods on SHD, SSC, and GSC-35 across varying network depths and parameter counts.

Figure 1: Coincidence detection: we consider two neurons $N_{1}$ and $N_{2}$ with the same positive synaptic weight values. $N_{2}$ has a delayed synaptic connection denoted $d_{21}$ of $8$ ms, thus both spikes from spike train $S_{1}$ and $S_{2}$ will reach $N_{2}$ quasi-simultaneously. As a result

Experimental results

Research questions

• RQ1 Can delays in deep feedforward SNNs be learned jointly with synaptic weights using backpropagation?
• RQ2 Does learning delays provide a significant accuracy advantage over fixed or random delays on temporal spike-pattern benchmarks?
• RQ3 How does the DCLS-delays approach compare to dense delay representations in terms of performance and parameter efficiency?
• RQ4 What is the impact of gradually tightening the Gaussian kernel (sigma) during training on learned delays and overall accuracy?

Key findings

• DCLS-Delays with 2–3 hidden layers achieves up to 95.07% on SHD, 79.77–80.69% on SSC, and 94.91–95.35% on GSC-35 under various configurations.
• Learned delays outperform fixed/random delays, especially in sparse connectivity scenarios.
• Dense convolution-delays with learnable spacings yields higher accuracy than standard dense delays and uses fewer parameters.
• The method attains state-of-the-art results on SHD, SSC, and GSC-35 without recurrent connections beyond LIF internal recurrence.
• Ablation studies show that jointly learning weights and delays with decreasing sigma improves performance over constant sigma or fixed delays.

Figure 2: Example of one neuron with 2 afferent synaptic connections, convolving $K1$ and $K2$ with the zero left-padded $S_{1}$ and $S_{2}$ is equivalent to following Equation 6