[Paper Review] ESL-SNNs: An Evolutionary Structure Learning Strategy for Spiking Neural Networks

Jiangrong Shen, Qi Xu+4 more|arXiv (Cornell University)|Jun 6, 2023

Advanced Memory and Neural Computing4 citations

TL;DR

This paper proposes ESL-SNNs, a biologically inspired evolutionary structure learning framework that enables sparse training of spiking neural networks from scratch by dynamically pruning and regenerating synaptic connections during training. The method achieves state-of-the-art accuracy with minimal performance loss—only 0.28% accuracy drop at 10% connection density—while significantly reducing memory and energy consumption, thereby closing the expressibility gap between sparse and dense training.

ABSTRACT

Spiking neural networks (SNNs) have manifested remarkable advantages in power consumption and event-driven property during the inference process. To take full advantage of low power consumption and improve the efficiency of these models further, the pruning methods have been explored to find sparse SNNs without redundancy connections after training. However, parameter redundancy still hinders the efficiency of SNNs during training. In the human brain, the rewiring process of neural networks is highly dynamic, while synaptic connections maintain relatively sparse during brain development. Inspired by this, here we propose an efficient evolutionary structure learning (ESL) framework for SNNs, named ESL-SNNs, to implement the sparse SNN training from scratch. The pruning and regeneration of synaptic connections in SNNs evolve dynamically during learning, yet keep the structural sparsity at a certain level. As a result, the ESL-SNNs can search for optimal sparse connectivity by exploring all possible parameters across time. Our experiments show that the proposed ESL-SNNs framework is able to learn SNNs with sparse structures effectively while reducing the limited accuracy. The ESL-SNNs achieve merely 0.28% accuracy loss with 10% connection density on the DVS-Cifar10 dataset. Our work presents a brand-new approach for sparse training of SNNs from scratch with biologically plausible evolutionary mechanisms, closing the gap in the expressibility between sparse training and dense training. Hence, it has great potential for SNN lightweight training and inference with low power consumption and small memory usage.

Motivation & Objective

• To address the high power consumption and memory usage of SNNs during training due to parameter redundancy.
• To develop a biologically plausible, sparse structure learning framework that enables genuine sparse training from scratch, not fine-tuning from dense models.
• To improve the expressibility of sparse SNN training by enabling dynamic exploration of synaptic connections across time.
• To reduce energy and memory overhead during both training and inference by maintaining structural sparsity from initialization.
• To close the performance gap between sparse and dense SNN training through evolutionary connectivity optimization.

Proposed method

• The ESL-SNNs framework initializes SNNs using an Erdős–Rényi random graph to establish a sparse baseline with fixed connection probability.
• During training, synaptic connections are dynamically pruned and regenerated every T_iter iterations based on predefined pruning and growth rules, maintaining a target structural sparsity level.
• Synaptic weights are updated via a global learning rule in each training iteration, while structural changes occur independently through evolutionary mechanisms inspired by brain synaptic rewiring.
• The framework supports various growth and pruning rules, including SET pruning and momentum-based growth, to guide the evolution of optimal sparse connectivity.
• The method enables end-to-end training of SNNs with sparse topology from initialization, avoiding the need for pre-training on dense architectures.
• The framework is compatible with multiple SNN architectures, including feedforward and convolutional networks, and supports different connection density levels.

Experimental results

Research questions

• RQ1 Can sparse SNN training from scratch be achieved using a biologically plausible evolutionary mechanism that mimics synaptic rewiring in the brain?
• RQ2 How does dynamic pruning and regeneration of synaptic connections during training affect the final model accuracy and sparsity level?
• RQ3 What is the impact of different pruning and growth rules on the stability and expressibility of sparse SNN training?
• RQ4 To what extent can ESL-SNNs reduce parameter count and energy consumption while maintaining competitive accuracy compared to dense SNNs?
• RQ5 Can the proposed framework close the performance gap between sparse and dense SNN training in terms of model expressibility?

Key findings

• ESL-SNNs achieve only 0.28% accuracy loss at 10% connection density on the DVS-Cifar10 dataset, demonstrating high efficiency and minimal performance degradation.
• At 50% connection density, ESL-SNNs attain 91.09% accuracy on Cifar10 and 73.48% on Cifar100 with approximately 6.28 million parameters, showing strong performance with reduced complexity.
• The model with SET pruning and momentum-based growth rules exhibits the most stable performance across different density levels, maintaining high accuracy even at 1% density (14.08% accuracy loss).
• The framework enables sparse training from scratch, significantly reducing memory usage and power consumption during both training and inference compared to dense SNNs.
• ESL-SNNs outperform or match state-of-the-art models like TET and LIAF-Net on DVS-Cifar10 with fewer connections and higher test accuracy at 78.3%.
• The dynamic evolution of synaptic connections allows for effective parameter space exploration, enhancing the expressibility of sparse training and enabling on-chip learning potential.