[Paper Review] Enabling Spike-based Backpropagation in State-of-the-art Deep Neural Network Architectures

Chankyu Lee, Syed Shakib Sarwar+1 more|arXiv (Cornell University)|Mar 15, 2019

Advanced Memory and Neural Computing32 references41 citations

TL;DR

This paper proposes a differentiable approximation of Leaky Integrate-and-Fire (LIF) neurons to enable end-to-end spike-based backpropagation in deep SNNs, achieving state-of-the-art accuracy on MNIST, SVHN, and CIFAR-10 using VGG and Residual architectures, while preserving temporal dynamics and enabling sparse, event-driven inference.

ABSTRACT

Spiking Neural Networks (SNNs) has recently emerged as a prominent neural computing paradigm. However, the typical shallow spiking network architectures have limited capacity for expressing complex representations, while training a very deep spiking network have not been successful so far. Diverse methods have been proposed to get around this issue such as converting off-line trained deep Artificial Neural Networks (ANNs) to SNNs. However, ANN-to-SNN conversion scheme fails to capture the temporal dynamics of a spiking system. On the other hand, it is still a difficult problem to directly train deep SNNs using input spike events due to the discontinuous and non-differentiable nature of the spike signals. To overcome this problem, we propose using differentiable (but approximate) activation for Leaky Integrate-and-Fire (LIF) spiking neurons to train deep convolutional SNNs with input spike events using spike-based backpropagation algorithm. Our experiments show the effectiveness of the proposed spike-based learning strategy on state-of-the-art deep networks (VGG and Residual architectures) by achieving the best classification accuracies in MNIST, SVHN and CIFAR-10 datasets compared to other SNNs trained with spike-based learning. Moreover, we analyze sparse event-driven computations to demonstrate the efficacy of proposed SNN training method for inference operation in the spiking domain.

Motivation & Objective

• To overcome the challenge of training deep Spiking Neural Networks (SNNs) with input spike events due to the non-differentiable nature of spike signals.
• To enable direct training of deep SNNs using spike-based backpropagation, preserving temporal dynamics lost in ANN-to-SNN conversion methods.
• To develop a differentiable approximation of LIF neurons that allows gradient flow through spike events in deep architectures.
• To evaluate the proposed method on state-of-the-art deep network structures (VGG and Residual) for image classification tasks.
• To demonstrate the efficiency of sparse, event-driven inference in the trained SNNs.

Proposed method

• Introduces a differentiable approximation of the spiking neuron's firing behavior in Leaky Integrate-and-Fire (LIF) neurons to allow gradient computation through spike events.
• Applies the spike-based backpropagation algorithm to train deep SNNs end-to-end using input spike trains.
• Employs the proposed differentiable LIF neuron in deep convolutional SNNs based on VGG and residual network architectures.
• Utilizes event-driven inference with sparse computation, leveraging the inherent sparsity of spike events for efficiency.
• Trains the SNNs directly on spike-encoded data without relying on pre-trained ANNs or conversion pipelines.
• Optimizes the network using standard backpropagation with gradients computed through the differentiable approximation of the spiking nonlinearity.

Experimental results

Research questions

• RQ1 Can deep SNNs be effectively trained using spike-based backpropagation with end-to-end learning on input spike events?
• RQ2 How does the proposed differentiable LIF approximation compare to existing SNN training methods in terms of classification accuracy?
• RQ3 Can the proposed method achieve state-of-the-art performance on standard image datasets like MNIST, SVHN, and CIFAR-10?
• RQ4 To what extent does the sparsity of spike events enable efficient inference in the trained SNNs?
• RQ5 Does the method preserve the temporal dynamics of spiking systems better than ANN-to-SNN conversion approaches?

Key findings

• The proposed method achieves the best classification accuracy among SNNs trained with spike-based learning on MNIST, SVHN, and CIFAR-10 datasets.
• The differentiable LIF approximation enables stable and effective backpropagation through spike events in deep SNNs.
• The method successfully trains deep SNNs based on VGG and residual architectures using input spike events.
• The trained SNNs support sparse, event-driven inference, demonstrating computational efficiency in the spiking domain.
• The approach outperforms existing spike-based training methods on benchmark image classification tasks.
• The results confirm that direct spike-based training with differentiable neurons can achieve high accuracy while maintaining temporal dynamics.