[Paper Review] Advancing Spatio-Temporal Processing in Spiking Neural Networks through Adaptation

Maximilian Baronig, R Ferrand+2 more|arXiv (Cornell University)|Aug 14, 2024

Advanced Memory and Neural Computing4 citations

TL;DR

This paper advances spiking neural networks (SNNs) by introducing a provably stable discretization method for adaptive LIF neurons with sub-threshold adaptation, significantly improving training stability and performance on spatio-temporal tasks. The method achieves state-of-the-art results on event-based benchmarks like SHD and BSD without requiring normalization, while demonstrating robustness to input shifts and strong generalization in time series prediction and generation.

ABSTRACT

Implementations of spiking neural networks on neuromorphic hardware promise orders of magnitude less power consumption than their non-spiking counterparts. The standard neuron model for spike-based computation on such systems has long been the leaky integrate-and-fire (LIF) neuron. A computationally light augmentation of the LIF neuron model with an adaptation mechanism has recently been shown to exhibit superior performance on spatio-temporal processing tasks. The root of the superiority of these so-called adaptive LIF neurons however is not well understood. In this article, we thoroughly analyze the dynamical, computational, and learning properties of adaptive LIF neurons and networks thereof. Our investigation reveals significant challenges related to stability and parameterization when employing the conventional Euler-Forward discretization for this class of models. We report a rigorous theoretical and empirical demonstration that these challenges can be effectively addressed by adopting an alternative discretization approach - the Symplectic Euler method, allowing to improve over state-of-the-art performances on common event-based benchmark datasets. Our further analysis of the computational properties of networks of adaptive LIF neurons shows that they are particularly well suited to exploit the spatio-temporal structure of input sequences without any normalization techniques.

Motivation & Objective

• To address the instability issues in training recurrent SNNs with adaptive LIF neurons, particularly those using sub-threshold adaptation.
• To understand the dynamical and computational advantages of adaptive LIF neurons over standard LIF neurons in spatio-temporal processing.
• To demonstrate that adaptive LIF networks can generalize robustly to input shifts in mean strength and spike rate without normalization.
• To achieve state-of-the-art performance on event-based datasets (SHD, BSD) using a novel, stable training method.
• To show that adaptive LIF networks excel in complex time series prediction and generation tasks, outperforming standard LIF and threshold-adaptation models.

Proposed method

• Proposes an alternative discretization method for adaptive LIF neurons that ensures provably better stability than the standard Euler-Forward method.
• Uses a continuous-time formulation of the adaptive LIF neuron with two state variables: membrane potential $u(t)$ and adaptation current $w(t)$, where $w(t)$ is updated by both spikes and sub-threshold membrane potential.
• Applies backpropagation through time (BPTT) to train recurrent SNNs with adaptive LIF neurons on spatio-temporal tasks.
• Employs a smoothing regularization technique during feature visualization: $X^{k+1} \leftarrow \nu X^{k+1} + \gamma(g * X^{k+1})$, with a Gaussian kernel $g(s)$ whose width $\sigma$ decays linearly over iterations.
• For data normalization in BSD, applies sample-wise normalization: $X^{k+1} \leftarrow X^{k+1} \cdot \mu_{\text{all}} / \mu_{k+1}$, where $\mu_{\text{all}}$ is the global mean spike rate.
• Uses a gradient clipping value of 1.5 and orthogonal initialization for recurrent weights, with feed-forward weights initialized uniformly.

Figure 1 : The adLIF neuron with membrane potential oscillation and spike-frequency adaptation. a) Neurons in the brain have been classified into two excitability classes, integrators (class 1) and resonators (class 2). While resonators show membrane potential oscillations, integrators do not. b) Me

Experimental results

Research questions

• RQ1 Why do adaptive LIF neurons with sub-threshold adaptation outperform standard LIF and threshold-adaptation models in spatio-temporal tasks?
• RQ2 What causes the instability in training adaptive LIF networks, and can it be resolved through improved discretization?
• RQ3 To what extent are adaptive LIF networks robust to shifts in input mean and spike rate, especially when such shifts were not seen during training?
• RQ4 Can adaptive LIF networks achieve state-of-the-art performance on event-based benchmarks like SHD and BSD without batch normalization or similar techniques?
• RQ5 How well do adaptive LIF networks generalize to complex time series prediction and generation tasks?

Key findings

• The proposed alternative discretization method for adaptive LIF neurons provides provably better stability than the Euler-Forward method, resolving frequent training instabilities.
• The method achieved a new state-of-the-art performance on the SHD and BSD event-based benchmark datasets, outperforming prior SNNs and even LSTM baselines.
• Adaptive LIF networks demonstrated strong generalization, remaining robust to shifts in input mean strength and spike rate—even when such shifts were not present during training.
• The networks achieved high performance on complex time series prediction and generation tasks, with sub-threshold adaptation enabling better exploitation of spatio-temporal input structure.
• The models achieved high accuracy without requiring normalization techniques such as batch normalization or batch-normalization through time, highlighting their inherent robustness.
• Feature visualization using adaptive smoothing revealed that the network learns meaningful, structured representations of input sequences, particularly in the SHD and BSD datasets.

Figure 2 : The adLIF neuron model shows frequency-selective resonating behavior. a) Voltage response (root mean squared membrane potential over $10$ seconds) of a LIF neuron in response to tonic spike input of different rates. b) Membrane potential response of an adLIF neuron with intrinsic frequenc