[Paper Review] Computing with hardware neurons: spiking or classical? Perspectives of applied Spiking Neural Networks from the hardware side

Sergei Dytckov, Masoud Daneshtalab|arXiv (Cornell University)|Feb 5, 2016

Advanced Memory and Neural Computing26 references3 citations

TL;DR

This paper compares spiking and classical neural networks from a hardware energy efficiency perspective, showing that while spiking hardware matches or exceeds classical systems in basic operations, system-level inefficiencies from excessive spike representation hinder performance. The key finding is that spike-driven applications minimizing spike counts could close the energy gap, but current systems remain bottlenecked by spike volume.

ABSTRACT

While classical neural networks take a position of a leading method in the machine learning community, spiking neuromorphic systems bring attention and large projects in neuroscience. Spiking neural networks were shown to be able to substitute networks of classical neurons in applied tasks. This work explores recent hardware designs focusing on perspective applications (like convolutional neural networks) for both neuron types from the energy efficiency side to analyse whether there is a possibility for spiking neuromorphic hardware to grow up for a wider use. Our comparison shows that spiking hardware is at least on the same level of energy efficiency or even higher than non-spiking on a level of basic operations. However, on a system level, spiking systems are outmatched and consume much more energy due to inefficient data representation with a long series of spikes. If spike-driven applications, minimizing an amount of spikes, are developed, spiking neural systems may reach the energy efficiency level of classical neural systems. However, in the near future, both type of neuromorphic systems may benefit from emerging memory technologies, minimizing the energy consumption of computation and memory for both neuron types. That would make infrastructure and data transfer energy dominant on the system level. We expect that spiking neurons have some benefits, which would allow achieving better energy results. Still the problem of an amount of spikes will still be the major bottleneck for spiking hardware systems.

Motivation & Objective

• To evaluate the energy efficiency of spiking neuromorphic hardware relative to classical neural networks in practical applications.
• To identify system-level inefficiencies in spiking hardware, particularly those arising from data representation via long spike trains.
• To explore whether spiking neural systems can achieve comparable or better energy efficiency than classical networks through optimized spike usage.
• To assess the role of emerging memory technologies in reducing energy consumption for both neuron types at the system level.
• To determine whether spiking neuromorphic systems can overcome current limitations to enable broader deployment.

Proposed method

• Benchmarking energy efficiency of spiking and classical neural networks at the level of basic computational operations.
• Analyzing system-level energy consumption, focusing on data representation inefficiencies in spiking systems due to prolonged spike sequences.
• Evaluating the impact of spike-driven application design on reducing overall spike count and improving energy efficiency.
• Assessing the potential of emerging memory technologies to reduce energy costs for both computation and memory in neuromorphic systems.
• Comparing system-level energy consumption trends under varying design choices, including spike frequency and data encoding strategies.

Experimental results

Research questions

• RQ1 How does the energy efficiency of spiking neuromorphic hardware compare to classical neural networks at the basic operation level?
• RQ2 What are the primary system-level energy bottlenecks in spiking neuromorphic systems, and how do they affect overall performance?
• RQ3 Can reducing spike count through application-level optimization close the energy efficiency gap between spiking and classical networks?
• RQ4 To what extent can emerging memory technologies reduce energy consumption in both spiking and classical neuromorphic systems?
• RQ5 What factors will determine whether spiking hardware can achieve system-level energy efficiency comparable to classical hardware?

Key findings

• Spiking hardware achieves energy efficiency comparable to or better than classical hardware at the level of basic operations.
• System-level energy consumption in spiking systems is significantly higher than in classical systems due to inefficient data representation via long spike trains.
• Applications that minimize spike counts could enable spiking neuromorphic systems to reach the energy efficiency of classical systems.
• Emerging memory technologies are expected to reduce computation and memory energy costs for both neuron types, shifting the dominant energy cost to infrastructure and data transfer.
• Despite potential advantages, the volume of spikes remains the primary bottleneck limiting the scalability and efficiency of spiking neuromorphic hardware.