[Paper Review] RMP-Loss: Regularizing Membrane Potential Distribution for Spiking Neural Networks

Yufei Guo, Xiaode Liu+6 more|arXiv (Cornell University)|Aug 13, 2023

Advanced Memory and Neural Computing4 citations

TL;DR

This paper proposes RMP-Loss, a simple yet effective regularization loss that redistributes membrane potential distributions in spiking neural networks (SNNs) toward binary spike values (0 or 1) to reduce quantization error. By directly minimizing the distance between membrane potentials and spike thresholds during training, RMP-Loss improves SNN accuracy without adding parameters or inference overhead, achieving state-of-the-art performance on CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet with fewer timesteps.

ABSTRACT

Spiking Neural Networks (SNNs) as one of the biology-inspired models have received much attention recently. It can significantly reduce energy consumption since they quantize the real-valued membrane potentials to 0/1 spikes to transmit information thus the multiplications of activations and weights can be replaced by additions when implemented on hardware. However, this quantization mechanism will inevitably introduce quantization error, thus causing catastrophic information loss. To address the quantization error problem, we propose a regularizing membrane potential loss (RMP-Loss) to adjust the distribution which is directly related to quantization error to a range close to the spikes. Our method is extremely simple to implement and straightforward to train an SNN. Furthermore, it is shown to consistently outperform previous state-of-the-art methods over different network architectures and datasets.

Motivation & Objective

• To directly address the quantization error in SNNs caused by mapping real-valued membrane potentials to binary spikes (0/1), which leads to information loss and performance degradation.
• To develop a training method that minimizes this quantization error without introducing additional parameters or computational overhead during inference.
• To improve SNN performance across diverse architectures and datasets by encouraging membrane potentials to concentrate near spike thresholds or reset values.
• To provide a theoretically grounded, simple, and universally applicable regularization technique that enhances training stability and accuracy in end-to-end SNN training.

Proposed method

• Introduce a novel regularization loss, RMP-Loss, which penalizes the L2 distance between membrane potentials and their corresponding 0/1 spike values during backpropagation.
• The loss function is added as a regularizer to the standard task loss, encouraging membrane potentials to cluster around binary spike levels (0 or 1) during training.
• The method operates directly on the membrane potential distribution, without modifying the network architecture or activation functions.
• RMP-Loss is differentiable and compatible with standard backpropagation, enabling end-to-end training of SNNs with surrogate gradient methods.
• The regularization is applied layer-wise and does not alter the inference mechanism or spike generation rules of the SNN.
• The approach is architecture-agnostic and can be easily integrated into any SNN training pipeline with minimal code changes.

Figure 1 : The overall workflow of the proposed method. We embed a membrane potential regularization loss in the task loss to redistribute the membrane potential in the training phase to reduce the quantization error.

Experimental results

Research questions

• RQ1 Can directly regularizing the distribution of membrane potentials toward binary spike values reduce quantization error in SNNs?
• RQ2 Does a simple, parameter-free regularization loss improve SNN accuracy without increasing inference cost?
• RQ3 How does RMP-Loss compare to state-of-the-art SNN training methods in terms of accuracy and efficiency across different datasets and architectures?
• RQ4 Can RMP-Loss achieve better performance than methods that indirectly reduce quantization error via architectural changes or increased timesteps?

Key findings

• On CIFAR-10, RMP-Loss achieves 93.65% top-1 accuracy with ResNet20 and 4 timesteps, outperforming the prior state-of-the-art TET by 4.26%.
• On CIFAR-100, RMP-Loss achieves 66.65% top-1 accuracy with ResNet20 and 4 timesteps, surpassing Diet-SNN by 2.58% and Real Spike by 0.05%.
• On the neuromorphic CIFAR10-DVS dataset, RMP-Loss achieves 76.20% top-1 accuracy with ResNet19 and 10 timesteps, outperforming InfLoR-SNN by 0.70% and RecDis-SNN by 3.78%.
• On ImageNet, RMP-Loss achieves 65.17% top-1 accuracy with ResNet34 and 4 timesteps, ranking as the highest among standard SNNs with binary spikes, though slightly below SEW ResNet due to its non-binary spike output.
• Visualization results confirm that RMP-Loss narrows the membrane potential distribution range, reducing the distance to spike thresholds and minimizing quantization error.
• The method consistently improves performance across multiple architectures (ResNet19, ResNet20, VGG16) and datasets, demonstrating broad applicability and robustness.

Figure 2 : The effect of RMP-Loss. The overall original membrane potential distribution (left) and the redistributed membrane potential distribution by RMP-Loss (right) of the first layer of the second block in ResNet20 on CIFAR-10 test sets.