[Paper Review] A Fully Spiking Hybrid Neural Network for Energy-Efficient Object Detection

Biswadeep Chakraborty, Xueyuan She+1 more|arXiv (Cornell University)|Apr 21, 2021

Advanced Memory and Neural Computing67 references49 citations

TL;DR

This paper proposes a Fully Spiking Hybrid Neural Network (FSHNN) for energy-efficient object detection by integrating unsupervised Spike Time-Dependent Plasticity (STDP) with backpropagation (STBP) in a spiking convolutional architecture. The FSHNN achieves 42.6% mAP on MSCOCO, outperforms DNN-based RetinaNet and spiking YOLO under noisy conditions and limited data, and demonstrates lower uncertainty and improved generalization through Monte Carlo dropout-based uncertainty estimation.

ABSTRACT

This paper proposes a Fully Spiking Hybrid Neural Network (FSHNN) for energy-efficient and robust object detection in resource-constrained platforms. The network architecture is based on Convolutional SNN using leaky-integrate-fire neuron models. The model combines unsupervised Spike Time-Dependent Plasticity (STDP) learning with back-propagation (STBP) learning methods and also uses Monte Carlo Dropout to get an estimate of the uncertainty error. FSHNN provides better accuracy compared to DNN based object detectors while being 150X energy-efficient. It also outperforms these object detectors, when subjected to noisy input data and less labeled training data with a lower uncertainty error.

Motivation & Objective

• To develop a fully spiking neural network for object detection that enables energy-efficient inference in resource-constrained platforms like drones and edge robots.
• To improve robustness and generalization in low-data and noisy input scenarios by integrating unsupervised STDP learning with supervised backpropagation.
• To quantify uncertainty in spiking object detectors using Monte Carlo dropout, enabling better discrimination between true and false positives.
• To demonstrate that hybrid learning in SNNs can achieve performance comparable or superior to DNNs while significantly improving energy efficiency.

Proposed method

• The FSHNN uses a RetinaNet-based backbone with spiking convolutional layers pre-trained via unsupervised STDP learning on low-level features.
• A hybrid training strategy combines STDP for initial feature learning with backpropagation for end-to-end fine-tuning, preserving spiking dynamics.
• The network employs DNN-to-SNN conversion with threshold balancing to convert the final DNN to a fully spiking network while keeping STDP-trained layers frozen.
• Monte Carlo dropout is adapted to estimate predictive uncertainty in FSHNN, enabling uncertainty-aware inference.
• Rate-based encoding is used for input representation, where analog activations are approximated by spike train rates over 300 time steps.
• Energy efficiency is estimated using FLOPS and hardware-specific MAC/AC energy costs, with SNN energy consumption scaled by time steps (T=300).

Experimental results

Research questions

• RQ1 Can a hybrid spiking network combining STDP and backpropagation achieve higher accuracy than standard DNN-based object detectors?
• RQ2 Does integrating STDP learning improve generalization and robustness under noisy inputs or limited labeled data?
• RQ3 Can uncertainty estimation via Monte Carlo dropout be effectively applied to spiking object detectors to improve reliability?
• RQ4 To what extent does a fully spiking architecture reduce energy consumption compared to DNNs in object detection?

Key findings

• FSHNN achieves a mean Average Precision (mAP) of 42.6% on the MSCOCO dataset, surpassing the 38.8% mAP of standard RetinaNet.
• With six STDP layers, FSHNN outperforms spiking YOLO, which reports only 26.24% mAP, despite using a more efficient architecture.
• FSHNN reduces uncertainty by 20-30% compared to RetinaNet under noisy input conditions, as measured via Monte Carlo dropout.
• The FSHNN model shows superior generalization, maintaining high performance when trained on limited labeled data, unlike standard DNNs.
• Energy efficiency analysis shows a theoretical 147.54x improvement in energy efficiency (EE) over DNNs, based on AC operations and reduced time steps (T=300).
• The ablation study confirms that the combination of STDP and backpropagation (FSHNN) yields the highest mAP (0.426) and AR@100 (56.2) on COCO-source-test, outperforming models with only STDP or only backpropagation.