[Paper Review] Training spiking multi-layer networks with surrogate gradients on an analog neuromorphic substrate

Benjamin Cramer, Sebastian Billaudelle+10 more|arXiv (Cornell University)|Jun 12, 2020

Advanced Memory and Neural Computing75 references21 citations

TL;DR

This paper presents a hardware-in-the-loop training method for spiking multi-layer networks using surrogate gradients on the analog BrainScales-2 neuromorphic chip. By executing forward passes on hardware and backpropagation in software, the approach achieves high accuracy (97.5% on MNIST) with low power (<300 mW) and high throughput (70 k patterns/sec), demonstrating efficient, sparse, and energy-aware spiking network processing on analog substrates.

ABSTRACT

Spiking neural networks are nature's solution for parallel information processing with high temporal precision at a low metabolic energy cost. To that end, biological neurons integrate inputs as an analog sum and communicate their outputs digitally as spikes, i.e., sparse binary events in time. These architectural principles can be mirrored effectively in analog neuromorphic hardware. Nevertheless, training spiking neural networks with sparse activity on hardware devices remains a major challenge. Primarily this is due to the lack of suitable training methods that take into account device-specific imperfections and operate at the level of individual spikes instead of firing rates. To tackle this issue, we developed a hardware-in-the-loop strategy to train multi-layer spiking networks using surrogate gradients on the analog BrainScales-2 chip. Specifically, we used the hardware to compute the forward pass of the network, while the backward pass was computed in software. We evaluated our approach on downscaled 16x16 versions of the MNIST and the fashion MNIST datasets in which spike latencies encoded pixel intensities. The analog neuromorphic substrate closely matched the performance of equivalently sized networks implemented in software. It is capable of processing 70 k patterns per second with a power consumption of less than 300 mW. Added activity regularization resulted in sparse network activity with about 20 spikes per input, at little to no reduction in classification performance. Thus, overall, our work demonstrates low-energy spiking network processing on an analog neuromorphic substrate and sets several new benchmarks for hardware systems in terms of classification accuracy, processing speed, and efficiency. Importantly, our work emphasizes the value of hardware-in-the-loop training and paves the way toward energy-efficient information processing on non-von-Neumann architectures.

Motivation & Objective

• Address the challenge of training spiking neural networks with sparse spike-based activity on analog neuromorphic hardware.
• Overcome the limitations of existing training methods that rely on firing rates and ignore device-specific imperfections.
• Enable end-to-end training of deep spiking networks directly on analog neuromorphic substrates by integrating hardware and software computation.
• Achieve high classification accuracy, low power consumption, and high processing speed in a non-von-Neumann, event-driven computing framework.

Proposed method

• Employ a hardware-in-the-loop training strategy where the forward pass is executed on the analog BrainScales-2 neuromorphic chip, while the backward pass is computed in software using surrogate gradients.
• Encode pixel intensities in MNIST and fashion MNIST datasets as spike latencies to enable temporal encoding in spiking networks.
• Use surrogate gradients to backpropagate error signals through non-differentiable spike-generating neurons, enabling end-to-end training despite the discrete nature of spikes.
• Apply activity regularization to enforce sparse network activity, reducing average spike counts to about 20 per input without degrading performance.
• Train multi-layer spiking networks using backpropagation through time with surrogate gradients, adapted to the analog hardware constraints.
• Evaluate performance on downscaled 16x16 versions of MNIST and fashion MNIST to reduce computational load while preserving classification fidelity.

Experimental results

Research questions

• RQ1 Can surrogate gradient backpropagation be effectively applied to train deep spiking neural networks on an analog neuromorphic substrate with device-specific imperfections?
• RQ2 To what extent does hardware-in-the-loop training preserve classification accuracy compared to software-implemented equivalents on the same dataset?
• RQ3 How efficiently can the analog neuromorphic hardware process spiking patterns in terms of speed and power consumption?
• RQ4 Can activity regularization achieve sparse spike activity (e.g., ~20 spikes per input) without compromising classification performance on the MNIST and fashion MNIST datasets?
• RQ5 What is the trade-off between energy efficiency, processing speed, and accuracy when training spiking networks directly on analog neuromorphic hardware?

Key findings

• The hardware-in-the-loop training approach achieved a classification accuracy of 97.5% on the 16x16 MNIST dataset, matching the performance of software-implemented networks.
• The system processed 70,000 patterns per second with a power consumption of less than 300 mW, demonstrating high processing efficiency.
• Activity regularization successfully reduced average network activity to approximately 20 spikes per input while maintaining high classification accuracy.
• The analog neuromorphic substrate closely matched the performance of software-based equivalents, validating its suitability for training spiking networks.
• The approach sets new benchmarks for classification accuracy, processing speed, and energy efficiency in hardware-based spiking neural network systems.
• The results demonstrate the feasibility of training deep spiking networks directly on analog neuromorphic hardware using surrogate gradients, paving the way for energy-efficient, non-von-Neumann computing.