[Paper Review] Efficient Event-Based Object Detection: A Hybrid Neural Network with Spatial and Temporal Attention

Soikat Hasan Ahmed, Jan Finkbeiner+1 more|arXiv (Cornell University)|Mar 15, 2024

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a hybrid SNN-ANN backbone with spatial and temporal attention for efficient event-based object detection, leveraging spiking neural networks for low-power, high-temporal-resolution preprocessing and artificial neural networks for high-accuracy feature learning. The method achieves state-of-the-art performance comparable to pure ANN baselines while significantly reducing parameter and energy budgets, validated through hardware deployment on Intel Loihi 2 with sub-real-time inference at 1.7W.

ABSTRACT

Event cameras offer high temporal resolution and dynamic range with minimal motion blur, making them promising for robust object detection. While Spiking Neural Networks (SNNs) on neuromorphic hardware are often considered for energy-efficient and low latency event-based data processing, they often fall short of Artificial Neural Networks (ANNs) in accuracy and flexibility. Here, we introduce Attention-based Hybrid SNN-ANN backbones for event-based object detection to leverage the strengths of both SNN and ANN architectures. A novel Attention-based SNN-ANN bridge module captures sparse spatial and temporal relations from the SNN layer and converts them into dense feature maps for the ANN part of the backbone. Additionally, we present a variant that integrates DWConvL-STMs to the ANN blocks to capture slower dynamics. This multi-timescale network combines fast SNN processing for short timesteps with long-term dense RNN processing, effectively capturing both fast and slow dynamics. Experimental results demonstrate that our proposed method surpasses SNN-based approaches by significant margins, with results comparable to existing ANN and RNN-based methods. Unlike ANN-only networks, the hybrid setup allows us to implement the SNN blocks on digital neuromorphic hardware to investigate the feasibility of our approach. Extensive ablation studies and implementation on neuromorphic hardware confirm the effectiveness of our proposed modules and architectural choices. Our hybrid SNN-ANN architectures pave the way for ANN-like performance at a drastically reduced parameter, latency, and power budget.

Motivation & Objective

• To address the challenge of efficiently processing sparse, high-temporal-resolution event data for object detection while maintaining high accuracy.
• To combine the energy efficiency of spiking neural networks (SNNs) with the superior learning capacity of artificial neural networks (ANNs).
• To design a novel attention-based bridge module that effectively transforms sparse SNN features into dense, high-level representations for ANN processing.
• To demonstrate hardware feasibility and energy efficiency by implementing the SNN backbone on digital neuromorphic hardware (Intel Loihi 2).
• To achieve ANN-level performance with drastically reduced parameter and power budgets through a hybrid SNN-ANN architecture.

Proposed method

• The proposed architecture uses a hybrid SNN-ANN backbone, where SNN blocks process raw event data with high temporal resolution and low power.
• An attention-based SNN-ANN bridge module converts sparse SNN outputs into dense feature maps for the ANN part, using two components: Event-Rate Spatial (ERS) attention and Spatial-Aware Temporal (SAT) attention.
• ERS attention highlights spatial regions with high event activity, improving spatial feature representation.
• SAT attention models temporal dependencies in sparse event streams by attending to irregular spatio-temporal patterns across time.
• The network is trained end-to-end using surrogate gradient backpropagation, enabling error backpropagation through the SNN components.
• The SNN blocks are quantized to int8 and fused with batch normalization and LIF neuron dynamics for deployment on Intel Loihi 2, preserving accuracy and enabling low-latency inference.

Experimental results

Research questions

• RQ1 Can a hybrid SNN-ANN architecture achieve ANN-level performance in event-based object detection while maintaining SNN advantages in energy efficiency and temporal resolution?
• RQ2 How effective is the proposed attention-based SNN-ANN bridge module in capturing sparse spatial and temporal relations from event data?
• RQ3 What is the optimal balance between SNN and ANN block counts for accuracy and computational efficiency?
• RQ4 Can the SNN backbone be efficiently deployed on neuromorphic hardware with minimal accuracy loss and sub-real-time latency?
• RQ5 What are the power and latency characteristics of the hybrid architecture when implemented on digital neuromorphic chips like Loihi 2?

Key findings

• The proposed hybrid SNN-ANN model achieves mAP(.5) of 0.61, mAP(.75) of 0.34, and mAP(.5:.05:.95) of 0.35, surpassing baseline hybrid and SNN-only methods and matching state-of-the-art ANN-based approaches.
• The 4-4 configuration (four SNN and four ANN blocks) achieved optimal balance, matching the performance of the 3-5 setup while reducing computational overhead.
• The SNN blocks were successfully implemented on Intel Loihi 2, achieving a power consumption of 1.7W ± 0.1W and latency of 1.9ms ± 0.8ms per step, faster than real-time (5ms per step).
• Quantization to int8 with weight scaling and input fusion preserved accuracy, with mAP(.5) of 0.349 (int8) vs. 0.348 (float16), demonstrating hardware compatibility.
• The ablation study confirmed that both ERS and SAT attention modules significantly improve performance, with SAT attention being particularly effective in handling sparse event inputs.
• The hardware implementation demonstrates that the SNN backbone enables sub-real-time, low-power inference, outperforming commercial edge devices like the NVIDIA Jetson Orin Nano in power efficiency.