[Paper Review] SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking

Xingrun Xing, Boyan Gao+6 more|arXiv (Cornell University)|Jul 5, 2024

Ferroelectric and Negative Capacitance Devices4 citations

TL;DR

SpikeLLM introduces the first spiking large language model (7–70B parameters) by integrating saliency-driven spiking dynamics into LLMs, using a novel Optimal Brain Spiking framework for efficient, high-accuracy quantization. It achieves state-of-the-art performance in both OmniQuant and GPTQ pipelines, reducing WikiText2 perplexity by 25.51% and outperforming PB-LLM in accuracy and sparsity while enabling fully additive linear layers via ternary spiking neurons.

ABSTRACT

Recent advancements in large language models (LLMs) with billions of parameters have improved performance in various applications, but their inference processes demand significant energy and computational resources. In contrast, the human brain, with approximately 86 billion neurons, is much more energy-efficient than LLMs with similar parameters. Inspired by this, we redesign 7$\sim$70 billion parameter LLMs using bio-plausible spiking mechanisms, emulating the efficient behavior of the human brain. We propose the first spiking large language model, SpikeLLM. Coupled with the proposed model, two essential approaches are proposed to improve spike training efficiency: Generalized Integrate-and-Fire (GIF) neurons to compress spike length from $T$ to $\frac{T}{L} \log_2 L$ bits, and an Optimal Brain Spiking framework to divide outlier channels and allocate different $T$ for GIF neurons, which further compresses spike length to approximate $log_2T$ bits. The necessity of spike-driven LLM is proved by comparison with quantized LLMs with similar operations. In the OmniQuant pipeline, SpikeLLM reduces 11.01% WikiText2 perplexity and improves 2.55% accuracy of common scene reasoning on a LLAMA-7B W4A4 model. In the GPTQ pipeline, SpikeLLM achieves direct additive in linear layers, significantly exceeding PB-LLMs.

Motivation & Objective

• To address the high energy and computational costs of large language models (LLMs) by emulating the brain's energy efficiency through spiking neural networks (SNNs).
• To overcome the bottlenecks in SNNs—firing rate encoding inefficiency and non-differentiable spiking dynamics—when scaling to LLMs.
• To enable large-scale SNNs to match or exceed the performance of quantized ANNs while maintaining event-driven, sparse computation.
• To develop a plug-and-play quantization framework that integrates with mainstream post-training quantization pipelines like OmniQuant and GPTQ.
• To achieve fully additive linear layers in SNNs using ternary spiking neurons, enabling efficient, sparse, and high-accuracy inference.

Proposed method

• Proposes Generalized Integrate-and-Fire (GIF) neurons that auto-regressively encode salient values over multiple spiking steps, improving semantic expressivity over standard rate coding.
• Introduces Optimal Brain Spiking—a saliency-driven quantization framework that uses first-order gradients for activation channel selection and second-order Hessian approximations for weight channel selection.
• Applies per-channel salient outlier expansion, where salient channels are allocated more spiking steps (T steps), while non-salient ones use one or few steps.
• Integrates the framework into mainstream quantization pipelines: OmniQuant (4A4W) and GPTQ (weight-only quantization), enabling compatibility with existing training workflows.
• Employs ternary GIF neurons to achieve fully additive linear layers, replacing mixed-precision quantization with event-driven spiking for improved efficiency and accuracy.
• Uses equivalent spiking steps as a metric to compare SNNs and BNNs, ensuring fair evaluation of computational cost and sparsity.

Experimental results

Research questions

• RQ1 Can spiking neural networks be scaled to 7–70 billion parameter LLMs while maintaining high performance and energy efficiency?
• RQ2 How can saliency-based spiking improve semantic expressivity and generalization in low-bit quantized LLMs compared to traditional quantization?
• RQ3 Can spiking dynamics enable fully additive linear layers in SNNs, matching or exceeding the performance of binary neural networks like PB-LLM?
• RQ4 How does the Optimal Brain Spiking framework compare to random spiking step allocation in terms of accuracy and robustness?
• RQ5 To what extent can saliency-driven spiking reduce energy consumption and inference latency in large-scale LLMs?

Key findings

• SpikeLLM reduces WikiText2 perplexity by 25.51% compared to the OmniQuant-4A4W baseline on LLAMA2-7B, demonstrating significant performance gains in sequence modeling.
• In zero-shot evaluation across 6 datasets, SpikeLLM improves average accuracy by 3.08% over the OmniQuant-4A4W baseline.
• SpikeLLM achieves fully additive linear layers via ternary GIF neurons, enabling sparse, event-driven computation without mixed-precision quantization.
• On the PB-LLM comparison, SpikeLLM with 100% ACs and 1.25 equivalent steps achieves 52.10% average accuracy, outperforming PB-LLM at 95% ACs and 1.35 equivalent steps.
• The Optimal Brain Spiking framework significantly outperforms random spiking step allocation in both activation and weight channels, proving the effectiveness of saliency-based allocation.
• SpikeLLM with 70:25:5 spiking step distribution achieves 65.83% PIQA and 68.47% ARC-e accuracy, showing strong zero-shot generalization.