[Paper Review] Deep Residual Learning in Spiking Neural Networks

Wei Fang, Zhaofei Yu+4 more|arXiv (Cornell University)|Feb 8, 2021

Advanced Memory and Neural Computing66 references224 citations

TL;DR

Introduces Spike-Element-Wise (SEW) ResNet to enable deep, directly trained spiking neural networks, overcoming degradation and vanishing/exploding gradients, and achieving state-of-the-art results among directly trained SNNs on ImageNet, DVS Gesture, and CIFAR10-DVS.

ABSTRACT

Deep Spiking Neural Networks (SNNs) present optimization difficulties for gradient-based approaches due to discrete binary activation and complex spatial-temporal dynamics. Considering the huge success of ResNet in deep learning, it would be natural to train deep SNNs with residual learning. Previous Spiking ResNet mimics the standard residual block in ANNs and simply replaces ReLU activation layers with spiking neurons, which suffers the degradation problem and can hardly implement residual learning. In this paper, we propose the spike-element-wise (SEW) ResNet to realize residual learning in deep SNNs. We prove that the SEW ResNet can easily implement identity mapping and overcome the vanishing/exploding gradient problems of Spiking ResNet. We evaluate our SEW ResNet on ImageNet, DVS Gesture, and CIFAR10-DVS datasets, and show that SEW ResNet outperforms the state-of-the-art directly trained SNNs in both accuracy and time-steps. Moreover, SEW ResNet can achieve higher performance by simply adding more layers, providing a simple method to train deep SNNs. To our best knowledge, this is the first time that directly training deep SNNs with more than 100 layers becomes possible. Our codes are available at https://github.com/fangwei123456/Spike-Element-Wise-ResNet.

Motivation & Objective

• Motivate the need for deep, efficiently trainable spiking neural networks (SNNs) beyond shallow architectures.
• Analyze limitations of existing Spiking ResNet in realizing identity mappings and stable gradients.
• Propose Spike-Element-Wise (SEW) residual blocks to enable true residual learning in SNNs.
• Demonstrate that SEW ResNet scales to >100 layers and outperforms state-of-the-art directly trained SNNs on multiple datasets.

Proposed method

• Define a unified spiking neuron model with discrete-time dynamics (Eqs. 1-3) and surrogate gradient for backpropagation.
• Critically analyze Spiking ResNet and show it cannot easily implement identity mapping and suffers vanishing/exploding gradients (Eq. 8).
• Introduce SEW residual blocks where the residual and input are combined via an element-wise function g (ADD, AND, IAND) after a spike activation, enabling easy identity mapping (Eq. 9).
• Explain how SEW blocks prevent gradient degradation by ensuring constant or controlled gradient propagation (Eq. 11).
• Provide downsampling SEW blocks using SN-enabled shortcuts to preserve gradient flow across depth.

Experimental results

Research questions

• RQ1 Can residual learning be effectively realized in directly trained deep SNNs without resorting to ANN-to-SNN conversion?
• RQ2 Do SEW residual blocks allow stable training and identity mappings across very deep SNNs?
• RQ3 How do SEW ResNets perform on ImageNet and neuromorphic datasets compared to Spiking ResNets and existing SNN methods?
• RQ4 What is the effect of different element-wise functions g (ADD, AND, IAND) on learning dynamics and performance?

Key findings

• SEW ResNet overcomes the degradation problem observed in Spiking ResNet, with deeper SEW networks achieving higher accuracy (e.g., ImageNet results).
• On ImageNet, SEW ResNet-34/50/101/152 outperformed Spiking ResNet counterparts in accuracy, while SEW ResNet-101/152 are the first reported SNNs with >100 layers for direct training.
• SEW residual blocks enable easy identity mapping and stable gradient flow, with gradient analyses showing mitigated vanishing/exploding gradients in deep SEW networks.
• SEW ResNet achieves strong results on neuromorphic datasets: DVS Gesture and CIFAR10-DVS, with ADD-based SEW blocks delivering the best accuracy in many cases and requiring few time-steps.
• Compared to state-of-the-art directly trained SNNs and ANN-to-SNN conversions, SEW ResNet achieves competitive or superior accuracy with notably lower latency and fewer parameters on several benchmarks.