[Paper Review] Sparse Spiking Gradient Descent

Nicolas Perez-Nieves, Dan F. M. Goodman|arXiv (Cornell University)|May 18, 2021

Advanced Memory and Neural Computing48 references4 citations

TL;DR

This paper introduces Sparse Spiking Gradient Descent, a novel backpropagation algorithm for Spiking Neural Networks (SNNs) that exploits the inherent spatiotemporal sparsity of spikes to accelerate training and reduce memory usage. By redefining the surrogate gradient to concentrate updates on active neurons, the method achieves up to 150× faster backward passes and 85% memory reduction on datasets like Fashion-MNIST, Neuromorphic-MNIST, and Spiking Heidelberg Digits, while maintaining state-of-the-art accuracy.

ABSTRACT

There is an increasing interest in emulating Spiking Neural Networks (SNNs) on neuromorphic computing devices due to their low energy consumption. Recent advances have allowed training SNNs to a point where they start to compete with traditional Artificial Neural Networks (ANNs) in terms of accuracy, while at the same time being energy efficient when run on neuromorphic hardware. However, the process of training SNNs is still based on dense tensor operations originally developed for ANNs which do not leverage the spatiotemporally sparse nature of SNNs. We present here the first sparse SNN backpropagation algorithm which achieves the same or better accuracy as current state of the art methods while being significantly faster and more memory efficient. We show the effectiveness of our method on real datasets of varying complexity (Fashion-MNIST, Neuromophic-MNIST and Spiking Heidelberg Digits) achieving a speedup in the backward pass of up to 150x, and 85% more memory efficient, without losing accuracy.

Motivation & Objective

• To address the inefficiency of dense tensor operations in training SNNs, which fail to exploit the spatiotemporal sparsity inherent in spiking activity.
• To reduce the computational and memory overhead of backpropagation in SNNs, which currently limits end-to-end training on neuromorphic hardware.
• To develop a sparse backpropagation method that maintains high accuracy while significantly improving training speed and efficiency.
• To demonstrate that sparse gradient computation is feasible and beneficial in SNNs using a novel surrogate gradient formulation.
• To motivate broader adoption of sparse operators in deep learning frameworks by showing measurable performance gains in SNN training.

Proposed method

• The method redefines the surrogate gradient function to be larger near the spike threshold, concentrating gradient updates on active neurons and naturally inducing sparsity in backpropagation.
• It formulates a three-factor Hebbian-like learning rule derived from the modified surrogate gradient, enabling efficient gradient computation only on neurons that fired spikes.
• The algorithm skips over 98% of gradient operations by focusing computation exclusively on active neurons and their connected weights during the backward pass.
• It uses a sparse computation graph that dynamically excludes inactive neurons and zero-gradient contributions, reducing both FLOPs and memory usage.
• The approach is implemented via custom sparse CUDA kernels for fully connected SNN layers, optimized to handle event-driven spiking activity.
• The method is compatible with existing SNN training pipelines and can be extended to convolutional and pruned SNN architectures.

Experimental results

Research questions

• RQ1 Can sparsity in spiking activity be leveraged to accelerate backpropagation in SNNs without sacrificing accuracy?
• RQ2 How much speedup and memory reduction can be achieved by restricting gradient computation to only active neurons in SNNs?
• RQ3 Can a modified surrogate gradient function naturally induce a sparse learning rule that mirrors biological plausibility and computational efficiency?
• RQ4 Is sparse backpropagation in SNNs feasible and performant on standard hardware, such as GPUs, using custom sparse kernels?
• RQ5 Does the proposed method outperform dense backpropagation in terms of training efficiency while maintaining state-of-the-art accuracy on real-world datasets?

Key findings

• The proposed sparse backpropagation method achieves up to 150× speedup in the backward pass compared to dense implementations on real datasets.
• Memory usage is reduced by up to 85% due to skipping over 98% of gradient operations on inactive neurons and edges.
• The method maintains or improves accuracy on Fashion-MNIST, Neuromorphic-MNIST, and Spiking Heidelberg Digits, matching or exceeding state-of-the-art results.
• The speedup is achieved without modifying the network architecture, by redefining the surrogate gradient to favor active neurons.
• The approach is compatible with existing SNN training frameworks and can be extended to convolutional and pruned SNNs.
• The study demonstrates that sparse tensor operations can significantly accelerate SNN training, challenging the assumption that they are not beneficial for deep learning.