[Paper Review] Efficient Computation in Adaptive Artificial Spiking Neural Networks

Davide Zambrano, Roeland Nusselder+2 more|arXiv (Cornell University)|Oct 13, 2017

Advanced Memory and Neural Computing22 references19 citations

TL;DR

This paper proposes Adaptive Spiking Neural Networks (AdSNNs) that use spike-time coding with adaptive thresholding to achieve high accuracy comparable to deep artificial neural networks (ANNs), while reducing firing rates by up to an order of magnitude. By modeling spike generation via a dynamic threshold based on fast adaptation, AdSNNs enable efficient, low-rate neural coding that matches ANN performance with biologically plausible spiking activity.

ABSTRACT

Artificial Neural Networks (ANNs) are bio-inspired models of neural computation that have proven highly effective. Still, ANNs lack a natural notion of time, and neural units in ANNs exchange analog values in a frame-based manner, a computationally and energetically inefficient form of communication. This contrasts sharply with biological neurons that communicate sparingly and efficiently using binary spikes. While artificial Spiking Neural Networks (SNNs) can be constructed by replacing the units of an ANN with spiking neurons, the current performance is far from that of deep ANNs on hard benchmarks and these SNNs use much higher firing rates compared to their biological counterparts, limiting their efficiency. Here we show how spiking neurons that employ an efficient form of neural coding can be used to construct SNNs that match high-performance ANNs and exceed state-of-the-art in SNNs on important benchmarks, while requiring much lower average firing rates. For this, we use spike-time coding based on the firing rate limiting adaptation phenomenon observed in biological spiking neurons. This phenomenon can be captured in adapting spiking neuron models, for which we derive the effective transfer function. Neural units in ANNs trained with this transfer function can be substituted directly with adaptive spiking neurons, and the resulting Adaptive SNNs (AdSNNs) can carry out inference in deep neural networks using up to an order of magnitude fewer spikes compared to previous SNNs. Adaptive spike-time coding additionally allows for the dynamic control of neural coding precision: we show how a simple model of arousal in AdSNNs further halves the average required firing rate and this notion naturally extends to other forms of attention. AdSNNs thus hold promise as a novel and efficient model for neural computation that naturally fits to temporally continuous and asynchronous applications.

Motivation & Objective

• To address the inefficiency of high firing rates in existing Spiking Neural Networks (SNNs), which contrast with the low 1–5 Hz rates in biological neurons.
• To develop a spiking neural network architecture that matches the performance of deep artificial neural networks (ANNs) while operating with significantly reduced spike rates.
• To enable dynamic control of coding precision through adaptive mechanisms, such as arousal-based modulation, to further reduce required firing rates.
• To derive an analytical transfer function for adaptive spiking neurons that allows direct substitution of ANN units with spiking neurons without retraining.

Proposed method

• Employing an Adaptive Spiking Neuron (ASN) model with spike-triggered adaptation, where the threshold increases multiplicatively after each spike via a decaying kernel.
• Using a spike-time coding scheme where the number of spikes encodes the input magnitude, with the firing rate dynamically controlled by parameters $ m_f $, $ heta_0 $, and spike height $ h $.
• Deriving an analytical expression for the effective transfer function $ f(S) $, which maps continuous activation $ S $ to a normalized output $ y(S) $, enabling direct replacement of ANN units with spiking neurons.
• Introducing a dynamic 'arousal' mechanism that increases the firing rate selectively on selected inputs, improving coding precision without global rate increase.
• Modeling postsynaptic potentials (PSPs) using exponentially decaying kernels $ au_ heta, au_ ho, au_ u $, enabling efficient computation via recursive dynamical systems.
• Calibrating the transfer function to ensure $ f(S) = 0 $ for $ S ≤ heta_0/2 $, with a correction term $ c $ to align the function with desired activation thresholds.

Experimental results

Research questions

• RQ1 Can adaptive spiking neurons with dynamic thresholding achieve performance comparable to deep ANNs while operating at much lower firing rates?
• RQ2 How can spike-time coding be designed to allow efficient, low-rate neural representation of continuous input values, mimicking biological coding efficiency?
• RQ3 Can the precision of neural coding be dynamically controlled via a mechanism analogous to arousal, and what impact does this have on firing rate and accuracy?
• RQ4 What analytical form does the effective transfer function of an adaptive spiking neuron take, and how can it be used to map ANN units directly to SNNs?
• RQ5 To what extent can the firing rate be reduced without sacrificing classification accuracy, and how does this depend on adaptation parameters and network design?

Key findings

• AdSNNs achieve classification accuracy on MNIST, CIFAR-10, and ImageNet benchmarks that match or exceed state-of-the-art SNNs, while using up to 10x fewer spikes on average.
• The average firing rate in AdSNNs is reduced to 1–5 Hz, aligning with biological neuron rates, compared to hundreds of Hz in prior SNNs.
• The introduction of an arousal mechanism that selectively increases firing rates on selected inputs reduces the average firing rate by an additional 50% without degrading performance.
• The derived transfer function $ f(S) $, based on spike timing and adaptation dynamics, enables direct substitution of ANN units with adaptive spiking neurons, preserving network performance.
• Changing the time constant $ au_ heta $ affects the shape of the transfer function and allows trade-offs between coding precision and spike efficiency, with longer $ au_ heta $ reducing required spike counts.
• On CIFAR-100 and LSVRC-2012, the arousal method improved classification accuracy after an initial transient drop, demonstrating the effectiveness of dynamic precision control.