[Paper Review] Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks

Shikuang Deng, Shi Gu|arXiv (Cornell University)|Feb 28, 2021

Advanced Memory and Neural Computing32 references100 citations

TL;DR

The paper analyzes conversion error from ANN to SNN and introduces a threshold-balanced, soft-reset based pipeline that achieves near-zero accuracy loss with dramatically reduced simulation length.

ABSTRACT

Spiking neural networks (SNNs) are biology-inspired artificial neural networks (ANNs) that comprise of spiking neurons to process asynchronous discrete signals. While more efficient in power consumption and inference speed on the neuromorphic hardware, SNNs are usually difficult to train directly from scratch with spikes due to the discreteness. As an alternative, many efforts have been devoted to converting conventional ANNs into SNNs by copying the weights from ANNs and adjusting the spiking threshold potential of neurons in SNNs. Researchers have designed new SNN architectures and conversion algorithms to diminish the conversion error. However, an effective conversion should address the difference between the SNN and ANN architectures with an efficient approximation \DSK{of} the loss function, which is missing in the field. In this work, we analyze the conversion error by recursive reduction to layer-wise summation and propose a novel strategic pipeline that transfers the weights to the target SNN by combining threshold balance and soft-reset mechanisms. This pipeline enables almost no accuracy loss between the converted SNNs and conventional ANNs with only $\sim1/10$ of the typical SNN simulation time. Our method is promising to get implanted onto embedded platforms with better support of SNNs with limited energy and memory.

Motivation & Objective

• Analyze how conversion error propagates from layer to layer when transforming ANNs to SNNs.
• Develop a layer-wise conversion strategy that minimizes activation-value discrepancies between ANN and SNN.
• Propose threshold balance and a shift mechanism to approximate spiking frequencies with short simulation lengths.
• Provide theoretical and experimental validation across CNN architectures on standard datasets.

Proposed method

• Derive forward propagation and show that SNN outputs are a clipped, discrete version of ANN activations.
• Decompose conversion error into layer-wise terms and show it can be minimized by controlling per-layer activation differences.
• Propose a threshold ReLU activation in the source ANN and align V_th with y_th to reduce information loss.
• Add a shift delta to the ANN ReLU to minimize squared activation difference, deriving delta ≈ V_th/(2T).
• Provide an algorithm for setting layer thresholds V_th and transferring weights with a small bias correction V_th/(2T).
• Validate the approach with CIFAR-10/100 and ImageNet using CIFAR-Net, VGG-16, and ResNet-20 architectures.

Experimental results

Research questions

• RQ1 How can conversion error from ANN to SNN be decomposed and minimized on a per-layer basis?
• RQ2 What role do threshold balancing and activation-shift play in reducing simulation length while preserving accuracy?
• RQ3 Can an optimal, short simulation-length pipeline achieve near-ANN accuracy across standard CNN architectures?
• RQ4 How does modifying the source ANN activation (threshold ReLU) affect conversion performance and generalization?

Key findings

• The conversion error can be approximately decomposed into a sum of per-layer activation-discrepancy terms.
• Using threshold ReLU and a shift delta of V_th/(2T) minimizes layer-wise activation differences, reducing overall conversion loss.
• The proposed pipeline achieves almost no accuracy loss with only about 1/10 of the typical SNN simulation time.
• Experiments show comparable or better accuracy and significantly shorter simulation lengths compared to several existing methods across CIFAR-10/100 and ImageNet tasks.
• Converted SNNs can even outperform the source ANN in some cases (e.g., ResNet-20 on CIFAR-10/100).
• The approach dramatically reduces required simulation steps (e.g., 16–32 steps for some networks) while maintaining high accuracy.