[Paper Review] Spike-inspired Rank Coding for Fast and Accurate Recurrent Neural Networks

Alan Jeffares, Qinghai Guo+2 more|arXiv (Cornell University)|Oct 6, 2021

Advanced Memory and Neural Computing4 citations

TL;DR

This paper introduces spike-inspired rank coding (RC) for conventional recurrent neural networks (RNNs), enabling early inference by halting computation after the first output neuron 'spikes'—triggered by a threshold-crossing event. The method achieves up to 99.19% accuracy on temporally-encoded MNIST with inference after just one time step, significantly improving speed-accuracy trade-offs over both standard LSTMs and SNNs with temporal coding.

ABSTRACT

Biological spiking neural networks (SNNs) can temporally encode information in their outputs, e.g. in the rank order in which neurons fire, whereas artificial neural networks (ANNs) conventionally do not. As a result, models of SNNs for neuromorphic computing are regarded as potentially more rapid and efficient than ANNs when dealing with temporal input. On the other hand, ANNs are simpler to train, and usually achieve superior performance. Here we show that temporal coding such as rank coding (RC) inspired by SNNs can also be applied to conventional ANNs such as LSTMs, and leads to computational savings and speedups. In our RC for ANNs, we apply backpropagation through time using the standard real-valued activations, but only from a strategically early time step of each sequential input example, decided by a threshold-crossing event. Learning then incorporates naturally also *when* to produce an output, without other changes to the model or the algorithm. Both the forward and the backward training pass can be significantly shortened by skipping the remaining input sequence after that first event. RC-training also significantly reduces time-to-insight during inference, with a minimal decrease in accuracy. The desired speed-accuracy trade-off is tunable by varying the threshold or a regularization parameter that rewards output entropy. We demonstrate these in two toy problems of sequence classification, and in a temporally-encoded MNIST dataset where our RC model achieves 99.19% accuracy after the first input time-step, outperforming the state of the art in temporal coding with SNNs, as well as in spoken-word classification of Google Speech Commands, outperforming non-RC-trained early inference with LSTMs.

Motivation & Objective

• To bridge the efficiency of spiking neural networks (SNNs) with the accuracy of conventional artificial neural networks (ANNs) by integrating spike-inspired temporal coding into standard RNNs.
• To reduce computational cost and inference time in sequential models without sacrificing accuracy, by dynamically truncating sequences after a threshold-crossing event in the output.
• To enable tunable speed-accuracy trade-offs in RNN inference through adjustable thresholds and a learnable entropy regularization parameter.
• To demonstrate that rank coding, a mechanism from SNNs, can be effectively adapted to standard backpropagation through time in ANNs, improving inference efficiency.
• To show that RC-training leads to faster time-to-insight during inference, especially in real-time applications like keyword recognition.

Proposed method

• The method applies standard backpropagation through time using real-valued activations, but only processes input sequences until the first output neuron crosses a predefined threshold, triggering early inference.
• The threshold-crossing event determines when computation is halted, effectively truncating the sequence and reducing both forward and backward pass duration.
• A regularization term based on output entropy is introduced to encourage earlier, more reliable predictions, improving the speed-accuracy trade-off.
• The model is trained end-to-end using standard RNNs (e.g., LSTMs), with no architectural changes—only the timing of inference is modified via the threshold mechanism.
• The threshold can be tuned at inference time to control speed, and the regularization parameter β is optimized during training to promote early, accurate outputs.
• The approach is evaluated on sequence classification tasks, including a temporally-encoded MNIST variant and Google Speech Commands, with performance compared to non-RC-trained models and SNN baselines.

Experimental results

Research questions

• RQ1 Can spike-inspired rank coding be effectively applied to standard artificial recurrent neural networks to accelerate inference?
• RQ2 Does RC-training lead to faster time-to-insight during inference while maintaining or improving accuracy?
• RQ3 Can the speed-accuracy trade-off be tuned independently via threshold adjustment and regularization, and how does this compare to conventional methods?
• RQ4 How does RC-trained RNNs perform on real-world temporal sequence tasks like spoken-word classification compared to SOTA SNNs and non-RC ANNs?
• RQ5 To what extent does RC-training improve inference efficiency without requiring architectural changes to standard RNNs?

Key findings

• The RC model achieved 99.19% accuracy on a temporally-encoded MNIST dataset after only one input time step, outperforming state-of-the-art SNNs using rank coding.
• In Google Speech Commands classification, the RC-trained LSTM achieved higher accuracy than non-RC-trained models at any given inference speed, demonstrating superior speed-accuracy trade-offs.
• RC-training enabled a significant reduction in time-to-insight during inference, with minimal accuracy loss, especially when combined with entropy regularization.
• The speed-accuracy trade-off was more tunable in RC-trained models: lowering the threshold reduced accuracy more steeply, but varying the regularization parameter β allowed for earlier inference with only a small or even slightly improved accuracy.
• The method achieved computational savings in both forward and backward passes by skipping remaining input steps after the first threshold-crossing event, without modifying the network architecture or training algorithm.
• The results demonstrate that integrating a single neuromorphic mechanism—rank coding—into standard RNNs can yield substantial efficiency gains while maintaining high performance, suggesting a pragmatic path toward neuromorphic computing.