[Paper Review] The fine line between dead neurons and sparsity in binarized spiking neural networks

Jason K. Eshraghian, Wei Lü|arXiv (Cornell University)|Jan 28, 2022

Advanced Memory and Neural ComputingEngineering18 citations

TL;DR

This paper proposes threshold annealing—a warm-up strategy that gradually increases the firing threshold in binarized spiking neural networks (BSNNs) to balance spike activity and dynamic range. By enabling early network activity and later high-precision state accumulation, it reduces dead neurons by 71% and achieves state-of-the-art accuracy on four datasets using only binarized weights.

ABSTRACT

Spiking neural networks can compensate for quantization error by encoding information either in the temporal domain, or by processing discretized quantities in hidden states of higher precision. In theory, a wide dynamic range state-space enables multiple binarized inputs to be accumulated together, thus improving the representational capacity of individual neurons. This may be achieved by increasing the firing threshold, but make it too high and sparse spike activity turns into no spike emission. In this paper, we propose the use of `threshold annealing' as a warm-up method for firing thresholds. We show it enables the propagation of spikes across multiple layers where neurons would otherwise cease to fire, and in doing so, achieve highly competitive results on four diverse datasets, despite using binarized weights. Source code is available at https://github.com/jeshraghian/snn-tha/

Motivation & Objective

• To address the trade-off between spike activity and dynamic range in binarized spiking neural networks (BSNNs), where high thresholds suppress firing and low thresholds limit representational capacity.
• To reduce the number of dead neurons—neurons that cease firing entirely—during training, which hinders gradient flow and slows convergence.
• To improve learning efficiency and accuracy in BSNNs without increasing weight precision, by dynamically adjusting the firing threshold during training.
• To enable effective backpropagation through sparsely active networks by maintaining sufficient early-layer activity while still leveraging high-precision state accumulation in later phases.

Proposed method

• Introduces threshold annealing: a warm-up technique that increases the firing threshold θ over training iterations, starting from a low value and gradually rising to a higher steady-state value.
• Uses a fixed threshold at inference time, derived from the final threshold reached during training, ensuring minimal memory overhead (one full-precision value per layer).
• Applies the method within a standard backpropagation framework, with the threshold update occurring outside the training loop to avoid gradient interference.
• Employs a dynamic range (DR) evolution strategy that mimics the natural progression of state-space precision, balancing early activity and late-stage accumulation.
• Leverages the spiking neuron model with leaky integrate-and-fire dynamics, where spike emission depends on whether the membrane potential u^j_t exceeds θ.
• Uses binarized weights w^ij ∈ {−1, 1} and binary activations z^i_t ∈ {0, 1}, with post-synaptic potentials x^j_t+1 ∈ {−1, 0, 1} to enable efficient computation and memory savings.

Experimental results

Research questions

• RQ1 Can a dynamic threshold adjustment strategy improve training convergence and accuracy in binarized spiking neural networks?
• RQ2 To what extent does threshold annealing reduce the number of dead neurons compared to fixed-threshold training in BSNNs?
• RQ3 Can a warm-up strategy that increases the threshold over time balance spike activity and dynamic range without requiring full-precision weights?
• RQ4 How does threshold annealing affect performance on diverse datasets with varying input sparsity and temporal dynamics?
• RQ5 Does threshold annealing enable competitive accuracy in BSNNs while maintaining low memory footprint and hardware efficiency?

Key findings

• Threshold annealing reduced the number of dead neurons by 71% on the SHD dataset compared to fixed-threshold BSNNs, significantly improving training convergence.
• On the DVS128 Gesture dataset, the proposed BSNN with threshold annealing achieved 91.3% accuracy using only 98 Kbits of binarized weights, outperforming a full-precision network (82.1%) with 8.4 Mbits.
• On the CIFAR-10 dataset, the method achieved 87.9% accuracy with just 12 Kbits of binarized weights, surpassing a full-precision NH model (87.3%) using 794 Kbits.
• The method achieved state-of-the-art performance on all four datasets tested—CIFAR-10, SVHN, DVS128 Gesture, and SHD—despite using only binarized weights and minimal memory overhead.
• The final threshold θ_γ was consistently within 0.1% of the steady-state threshold θ_∞, allowing direct use of θ_∞ at inference without retraining or calibration.
• The distribution of spiking activity improved significantly: with threshold annealing, only ~20% of neurons were dead in both first and final layers on SHD, compared to 55% and 90% in the fixed-threshold baseline.