[论文解读] KLIF: An optimized spiking neuron unit for tuning surrogate gradient slope and membrane potential

Chunming Jiang, Yilei Zhang|arXiv (Cornell University)|Feb 18, 2023

Advanced Memory and Neural Computing被引用 8

一句话总结

KLIF 在基于 k 的漏扫积分并终化神经元中引入可学习的缩放因子，在训练过程中动态调整代理梯度斜率和膜电位，从而在不增加额外计算成本的情况下提升 SNN 性能。

ABSTRACT

Spiking neural networks (SNNs) have attracted much attention due to their ability to process temporal information, low power consumption, and higher biological plausibility. However, it is still challenging to develop efficient and high-performing learning algorithms for SNNs. Methods like artificial neural network (ANN)-to-SNN conversion can transform ANNs to SNNs with slight performance loss, but it needs a long simulation to approximate the rate coding. Directly training SNN by spike-based backpropagation (BP) such as surrogate gradient approximation is more flexible. Yet now, the performance of SNNs is not competitive compared with ANNs. In this paper, we propose a novel k-based leaky Integrate-and-Fire (KLIF) neuron model to improve the learning ability of SNNs. Compared with the popular leaky integrate-and-fire (LIF) model, KLIF adds a learnable scaling factor to dynamically update the slope and width of the surrogate gradient curve during training and incorporates a ReLU activation function that selectively delivers membrane potential to spike firing and resetting. The proposed spiking unit is evaluated on both static MNIST, Fashion-MNIST, CIFAR-10 datasets, as well as neuromorphic N-MNIST, CIFAR10-DVS, and DVS128-Gesture datasets. Experiments indicate that KLIF performs much better than LIF without introducing additional computational cost and achieves state-of-the-art performance on these datasets with few time steps. Also, KLIF is believed to be more biological plausible than LIF. The good performance of KLIF can make it completely replace the role of LIF in SNN for various tasks.

研究动机与目标

• 通过解决代理梯度的局限性来提升脉冲神经网络（SNN）的学习效果。
• 提出一种新颖的基于 k 的 Leaky Integrate-and-Fire（KLIF）神经元，具有可学习的缩放因子。
• 结合基于 ReLU 的机制来控制膜电位的放电/重置动力学。
• 在静态和神经形成为主的数据集上评估 KLIF，以展示相较于 LIF 的性能提升且不增加计算成本。
• 评估 KLIF 相较于传统 LIF 神经元在生物学可 plausibility 与实际部署方面的优势与潜在好处。

提出的方法

• 引入 KLIF：一个基于 k 的 Leaky Integrate-and-Fire 神经元。
• 添加一个可学习的缩放因子，在训练过程中动态更新代理梯度斜率与宽度。
• 引入一个 ReLU 激活，选择性地将膜电位传递给尖峰放电与重置。
• 在 MNIST、Fashion-MNIST、CIFAR-10 以及神经形态数据集（N-MNIST、CIFAR10-DVS、DVS128-Gesture）上评估 KLIF。
• 在不增加计算成本的前提下，与 LIF 进行性能对比。

实验结果

研究问题

• RQ1 KLIF 中的可学习缩放因子在训练过程中是否能够自适应代理梯度斜率以改善 SNN 的优化？
• RQ2 KLIF 的基于 ReLU 的机制是否提升了膜电位处理及尖峰/重置动力学相较于 LIF？
• RQ3 KLIF 是否在静态和神经形态数据集上以较少的时间步实现了前沿性能？
• RQ4 KLIF 是否在保持或提升性能与效率的同时具有更高的生物学合理性？

主要发现

• KLIF 在静态与神经形态数据集上均优于 LIF 模型，且未增加额外计算成本。
• KLIF 在评估数据集上以较少的时间步实现了前沿性能。
• 该模型被认为在保持效率的同时比 LIF 更具生物学可 plausibility。