[Paper Review] GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks

Xingting Yao, Fanrong Li+2 more|arXiv (Cornell University)|Oct 25, 2022

Advanced Memory and Neural Computing45 citations

TL;DR

GLIF unifies multiple bio-features in LIF neurons via learnable gating to enlarge neuronal dynamics and improve SNN performance, with channel-wise parametric learning and end-to-end training.

ABSTRACT

Spiking Neural Networks (SNNs) have been studied over decades to incorporate their biological plausibility and leverage their promising energy efficiency. Throughout existing SNNs, the leaky integrate-and-fire (LIF) model is commonly adopted to formulate the spiking neuron and evolves into numerous variants with different biological features. However, most LIF-based neurons support only single biological feature in different neuronal behaviors, limiting their expressiveness and neuronal dynamic diversity. In this paper, we propose GLIF, a unified spiking neuron, to fuse different bio-features in different neuronal behaviors, enlarging the representation space of spiking neurons. In GLIF, gating factors, which are exploited to determine the proportion of the fused bio-features, are learnable during training. Combining all learnable membrane-related parameters, our method can make spiking neurons different and constantly changing, thus increasing the heterogeneity and adaptivity of spiking neurons. Extensive experiments on a variety of datasets demonstrate that our method obtains superior performance compared with other SNNs by simply changing their neuronal formulations to GLIF. In particular, we train a spiking ResNet-19 with GLIF and achieve $77.35\%$ top-1 accuracy with six time steps on CIFAR-100, which has advanced the state-of-the-art. Codes are available at \url{https://github.com/Ikarosy/Gated-LIF}.

Motivation & Objective

• Increase the representational diversity of spiking neurons by fusing multiple bio-features in LIF behaviors.
• Introduce learnable gating factors to balance dual bio-features in membrane leakage, accumulation, and spike initiation.
• Enable channel-wise parameter learning to enhance neuronal heterogeneity and adaptability during training.
• Demonstrate state-of-the-art performance of GLIF-based SNNs on static and neuromorphic datasets.
• Provide ablation studies to validate the contribution of gating, channel-wise parametrization, and feature fusion.

Proposed method

• Propose GLIF, a unified spiking neuron that fuses dual bio-features for leakage, accumulation, and spike initiation through gating units.
• Define L(t,l)=GLIF membrane leakage as a fusion of linear and exponential decay via G_alpha.
• Define I(t,l) as integration accumulation with a flexible/canonical coding pattern via G_beta.
• Define F(t,l) as spike initiation via a soft/hard reset fusion controlled by G_gamma.
• Make primitives and gating factors learnable with a channel-wise parameter-sharing scheme.
• Train GLIF-SNNs end-to-end using backpropagation through time with a surrogate gradient for the Heaviside activation.

Experimental results

Research questions

• RQ1 Can fusion of dual bio-features in the LIF model via gating units increase the expressive power and performance of SNNs?
• RQ2 Does learning gating factors and channel-wise membrane parameters improve heterogeneity and accuracy across datasets?
• RQ3 How does GLIF compare to simplex LIF variants and other parametric spiking neurons on static and neuromorphic data?
• RQ4 What is the impact of channel-wise sharing versus layer-wise sharing on model performance?
• RQ5 Are learnable gating factors essential for achieving the reported gains?

Key findings

• GLIF outperforms state-of-the-art on CIFAR-10/100 with standard architectures and limited time steps (e.g., 77.35% top-1 on CIFAR-100 with ResNet-19 at 6 steps).
• On ImageNet, GLIF improves ResNet-34/MS-ResNet-18 baselines (e.g., 69.09% validation accuracy for GLIF-ResNet-34 at 6 steps, higher than STBP-tdBN).
• GLIF achieves 78.10% accuracy on CIFAR10-DVS using 7B-wideNet with 16 steps, outperforming SEW+PLIF and related methods.
• Ablations show GLIF’s gating factors and channel-wise parametrization substantially beat all simplex LIF variants and layer-wise sharing.
• Learnable gating factors (vs static) and channel-wise sharing consistently improve performance across multiple experiments.