[Paper Review] Beyond Classification: Directly Training Spiking Neural Networks for Semantic Segmentation

Youngeun Kim, Joshua Chough+1 more|arXiv (Cornell University)|Oct 14, 2021

Advanced Memory and Neural Computing48 references4 citations

TL;DR

This paper proposes direct training of spiking neural networks (SNNs) for semantic segmentation using surrogate gradient backpropagation with Batch Normalization Through Time (BNTT), achieving higher accuracy and energy efficiency than ANN-SNN conversion. The method enables low-latency inference on both static images and event-based data, demonstrating over 2× energy efficiency gains on static datasets.

ABSTRACT

Spiking Neural Networks (SNNs) have recently emerged as the low-power alternative to Artificial Neural Networks (ANNs) because of their sparse, asynchronous, and binary event-driven processing. Due to their energy efficiency, SNNs have a high possibility of being deployed for real-world, resource-constrained systems such as autonomous vehicles and drones. However, owing to their non-differentiable and complex neuronal dynamics, most previous SNN optimization methods have been limited to image recognition. In this paper, we explore the SNN applications beyond classification and present semantic segmentation networks configured with spiking neurons. Specifically, we first investigate two representative SNN optimization techniques for recognition tasks (i.e., ANN-SNN conversion and surrogate gradient learning) on semantic segmentation datasets. We observe that, when converted from ANNs, SNNs suffer from high latency and low performance due to the spatial variance of features. Therefore, we directly train networks with surrogate gradient learning, resulting in lower latency and higher performance than ANN-SNN conversion. Moreover, we redesign two fundamental ANN segmentation architectures (i.e., Fully Convolutional Networks and DeepLab) for the SNN domain. We conduct experiments on two public semantic segmentation benchmarks including the PASCAL VOC2012 dataset and the DDD17 event-based dataset. In addition to showing the feasibility of SNNs for semantic segmentation, we show that SNNs can be more robust and energy-efficient compared to their ANN counterparts in this domain.

Motivation & Objective

• To investigate the feasibility of deploying Spiking Neural Networks (SNNs) in semantic segmentation, a task beyond traditional image classification.
• To address the limitations of ANN-SNN conversion in segmentation tasks, where high latency and poor performance arise due to spatial and channel-wise activation variance.
• To develop SNN-specific architectures (Spiking-DeepLab and Spiking-FCN) that are compatible with event-based data from Dynamic Vision Sensors (DVS).
• To demonstrate that directly trained SNNs can achieve higher energy efficiency and robustness compared to their ANN counterparts in semantic segmentation.

Proposed method

• Uses surrogate gradient learning to backpropagate through non-differentiable LIF neurons by approximating the gradient with a piece-wise linear function.
• Employs Batch Normalization Through Time (BNTT) to stabilize training and enable end-to-end learning of SNNs from scratch.
• Redesigns two standard ANN segmentation architectures—Fully Convolutional Networks (FCN) and DeepLab—into spiking equivalents (Spiking-FCN and Spiking-DeepLab).
• Computes energy efficiency using FLOPs scaled by spike rates and CMOS energy models, with MAC operations in ANNs requiring 4.6 pJ and AC in SNNs only 0.9 pJ.
• Applies spatial cross-entropy loss during training to optimize SNN weights for pixel-wise segmentation.
• Evaluates performance on both static image (PASCAL VOC2012) and event-stream (DDD17) datasets to assess robustness and efficiency.

Experimental results

Research questions

• RQ1 Can SNNs be effectively trained for semantic segmentation using direct backpropagation, rather than relying on ANN-SNN conversion?
• RQ2 How does the performance of directly trained SNNs compare to ANN-SNN converted SNNs in terms of accuracy and inference latency on segmentation tasks?
• RQ3 To what extent do SNNs improve energy efficiency in semantic segmentation, especially when processing event-based data from DVS cameras?
• RQ4 How do spike rates and temporal dynamics vary across layers in spiking segmentation networks, and how do they affect energy consumption?

Key findings

• ANN-SNN conversion fails to achieve reasonable performance on semantic segmentation tasks due to high spatial and channel-wise activation variance, requiring over 1,000 time-steps for convergence.
• Direct training with surrogate gradients and BNTT achieves state-of-the-art SNN performance on both PASCAL VOC2012 and DDD17 datasets with significantly fewer time-steps.
• Spiking-FCN and Spiking-DeepLab achieve 2.37× and 2.75× higher energy efficiency than their ANN counterparts on the static PASCAL VOC2012 dataset.
• On the event-based DDD17 dataset, SNNs show only 1.04× and 1.15× energy efficiency gain over ANNs, due to higher spike rates, but still demonstrate compatibility with neuromorphic data.
• Spike rates are highest in upsampling layers due to skip connections, which inject additional spikes during forward pass.
• Energy consumption in SNNs is dynamic and data-dependent, unlike ANNs, which consume fixed energy regardless of input.