[Paper Review] Deep Directly-Trained Spiking Neural Networks for Object Detection

Qiaoyi Su, Yuhong Chou+5 more|arXiv (Cornell University)|Jul 21, 2023

Advanced Memory and Neural Computing15 citations

TL;DR

The paper introduces EMS-YOLO, a directly trained deep spiking neural network for object detection, achieving competitive COCO performance with only 4 time steps and substantially lower energy than ANN-SNN conversions. It also demonstrates strong results on event-based Gen1 data with real-time inference.

ABSTRACT

Spiking neural networks (SNNs) are brain-inspired energy-efficient models that encode information in spatiotemporal dynamics. Recently, deep SNNs trained directly have shown great success in achieving high performance on classification tasks with very few time steps. However, how to design a directly-trained SNN for the regression task of object detection still remains a challenging problem. To address this problem, we propose EMS-YOLO, a novel directly-trained SNN framework for object detection, which is the first trial to train a deep SNN with surrogate gradients for object detection rather than ANN-SNN conversion strategies. Specifically, we design a full-spike residual block, EMS-ResNet, which can effectively extend the depth of the directly-trained SNN with low power consumption. Furthermore, we theoretically analyze and prove the EMS-ResNet could avoid gradient vanishing or exploding. The results demonstrate that our approach outperforms the state-of-the-art ANN-SNN conversion methods (at least 500 time steps) in extremely fewer time steps (only 4 time steps). It is shown that our model could achieve comparable performance to the ANN with the same architecture while consuming 5.83 times less energy on the frame-based COCO Dataset and the event-based Gen1 Dataset.

Motivation & Objective

• Motivate energy-efficient object detection and explore directly trained SNNs for regression tasks such as bounding-box prediction.
• Develop a deep, fully-spiking backbone and detection head that avoid non-spike computations to reduce energy use.
• Enable multi-scale feature extraction within a fully spiking framework suitable for frame-based and event-based data.

Proposed method

• Propose EMS-YOLO, a YOLO-based detector trained directly as an SNN using surrogate gradients.
• Introduce EMS-ResNet, a full-spike residual block to enable deep directly-trained SNNs without gradient vanishing/exploding.
• Use Energy-Efficient Membrane-Shortcut blocks to avoid MAC operations on shortcut paths and maintain spike-only computation.
• Adopt input representations for both frame-based and event-based data, including temporal binning of events.
• Analyze gradient behavior via Block Dynamical Isometry theory to justify deep trainability.
• Evaluate energy consumption with a model that counts AC/MAC operations and spike-based energy metrics.

Experimental results

Research questions

• RQ1 Can a fully spike-based, directly trained SNN achieve competitive object detection performance without ANN-SNN conversion?
• RQ2 How can residual blocks be designed to keep computation fully spiking and energy-efficient at depth?
• RQ3 What is the energy efficiency of a directly trained SNN detector on frame-based COCO data and event-based Gen1 data?
• RQ4 Does the EMS-ResNet enable deep training without gradient vanishing/exploding in practice?
• RQ5 How do time steps and network depth affect accuracy in a directly trained SNN detector?

Key findings

• EMS-YOLO with EMS-ResNet34 achieves competitive performance on COCO2017 with only 4 time steps, outperforming some ANN-SNN conversion baselines that require hundreds to thousands of steps.
• The fully-spiking EMS-ResNet design eliminates non-spike MACs on shortcut paths, yielding substantial energy savings (reported up to 5.83x fewer energy than a comparable ANN).
• On the Gen1 event-based dataset, EMS-ResNet18/ResNet10 variants achieve higher mAP@0.5 and lower firing rates than comparable ANN or converted SNN baselines, indicating strong performance with sparse spikes.
• The EMS-ResNet backbone satisfies a gradient-dynamics condition (Block Dynamical Isometry) to mitigate vanishing/exploding gradients, enabling deeper directly-trained SNNs.
• Ablations show EMS-ResNet blocks provide energy efficiency and maintain performance across varying depths and time steps.