[Paper Review] Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies

Wei Fang, Zhaofei Yu+6 more|arXiv (Cornell University)|Apr 25, 2023

Advanced Memory and Neural ComputingEngineering15 citations

TL;DR

Introduces Parallel Spiking Neuron (PSN) families that remove reset to enable fully parallelized spiking dynamics, achieving faster simulation and improved long-term dependency learning; includes masked and sliding variants for variable-length sequences and step-by-step processing.

ABSTRACT

Vanilla spiking neurons in Spiking Neural Networks (SNNs) use charge-fire-reset neuronal dynamics, which can only be simulated serially and can hardly learn long-time dependencies. We find that when removing reset, the neuronal dynamics can be reformulated in a non-iterative form and parallelized. By rewriting neuronal dynamics without reset to a general formulation, we propose the Parallel Spiking Neuron (PSN), which generates hidden states that are independent of their predecessors, resulting in parallelizable neuronal dynamics and extremely high simulation speed. The weights of inputs in the PSN are fully connected, which maximizes the utilization of temporal information. To avoid the use of future inputs for step-by-step inference, the weights of the PSN can be masked, resulting in the masked PSN. By sharing weights across time-steps based on the masked PSN, the sliding PSN is proposed to handle sequences of varying lengths. We evaluate the PSN family on simulation speed and temporal/static data classification, and the results show the overwhelming advantage of the PSN family in efficiency and accuracy. To the best of our knowledge, this is the first study about parallelizing spiking neurons and can be a cornerstone for the spiking deep learning research. Our codes are available at \url{https://github.com/fangwei123456/Parallel-Spiking-Neuron}.

Motivation & Objective

• Motivate removing reset in spiking neuron dynamics to enable parallel computation.
• Develop a general PSN framework that yields non-iterative hidden state computation.
• Introduce masked PSN and sliding PSN to balance parallelism with step-wise inference and variable sequence lengths.
• Evaluate PSN variants on simulation speed, temporal and static data classification, and neuromorphic benchmarks.

Proposed method

• Reformulate spiking neuron charging without reset as a linear combination of inputs: H[t] = sum_i W_{t,i} X[i].
• Define PSN with H = W X and S = Theta(H - B), enabling fully parallel computation.
• Introduce k-order masked PSN by applying a time-local mask M_k to restrict dependence to the latest k inputs.
• Propose k-order sliding PSN with time-shared weights, equivalent to a 1D convolution over the input sequence.
• Provide training with Backpropagation Through Time (BPTT) while preserving parallel forward dynamics.
• Compare PSN variants against vanilla LIF/GLIF/KLIF/PLIF neurons on speed and learning long-term dependencies.

Figure 1: The computational graphs of the vanilla spiking neuron and the parallel spiking neuron. Figure (a) is cited from [ 14 ] . The dotted lines in Figure (b) are weights that can be masked for step-by-step computation. $X[t],S[t]$ are the input current and the output spike, and $H[t],V[t]$ are

Experimental results

Research questions

• RQ1 Can removing the reset operation enable fully parallel neuron dynamics without collapsing firing behavior?
• RQ2 How does the PSN family perform in simulation speed versus traditional spiking neurons?
• RQ3 Do masked and sliding PSN variants preserve step-by-step inference while improving sequence-length handling and accuracy?
• RQ4 Are PSN variants capable of learning long-term dependencies better than traditional spiking neurons on sequential and static tasks?
• RQ5 What is the impact of high-order parallelization on parameter efficiency and memory usage?

Key findings

• PSN achieves substantially faster simulation than vanilla LIF/IF neurons due to parallel matrix-multiplication-based dynamics.
• PSN variants (masked, sliding) maintain high accuracy on sequential CIFAR tasks, outperforming several traditional spiking neurons.
• Sequential CIFAR results show PSN bests; masked PSN and sliding PSN closely follow PSN with strong step-wise capability.
• On static and neuromorphic datasets, PSN and sliding PSN deliver competitive accuracy with fewer time-steps and parameters.
• Long-term dependency tasks (Sequential CIFAR-10/100) favor PSN-family over GLIF, KLIF, PLIF, and LIF variants (with reset or without).
• Sliding PSN generally achieves higher accuracy and better generalization than masked PSN across tasks."
• PSN adds negligible parameters in common SNN backbones (e.g., ResNet-18/ VGG-11) while delivering large throughput gains.

Figure 2: Comparison of the PSN family on the parameter number $n_{param}$ and the generation of hidden states.