[Paper Review] SpikePoint: An Efficient Point-based Spiking Neural Network for Event Cameras Action Recognition

Hongwei Ren, Yue Zhou+5 more|arXiv (Cornell University)|Oct 11, 2023

Advanced Memory and Neural ComputingEngineering3 citations

TL;DR

SpikePoint proposes a novel, end-to-end point-based spiking neural network that directly processes sparse event cloud data from event cameras without frame conversion, leveraging a singular-stage architecture with identity mapping and surrogate training to achieve state-of-the-art action recognition accuracy with only 16 timesteps, 0.3% of ANNs' parameters, and 0.5% of their power consumption across five datasets.

ABSTRACT

Event cameras are bio-inspired sensors that respond to local changes in light intensity and feature low latency, high energy efficiency, and high dynamic range. Meanwhile, Spiking Neural Networks (SNNs) have gained significant attention due to their remarkable efficiency and fault tolerance. By synergistically harnessing the energy efficiency inherent in event cameras and the spike-based processing capabilities of SNNs, their integration could enable ultra-low-power application scenarios, such as action recognition tasks. However, existing approaches often entail converting asynchronous events into conventional frames, leading to additional data mapping efforts and a loss of sparsity, contradicting the design concept of SNNs and event cameras. To address this challenge, we propose SpikePoint, a novel end-to-end point-based SNN architecture. SpikePoint excels at processing sparse event cloud data, effectively extracting both global and local features through a singular-stage structure. Leveraging the surrogate training method, SpikePoint achieves high accuracy with few parameters and maintains low power consumption, specifically employing the identity mapping feature extractor on diverse datasets. SpikePoint achieves state-of-the-art (SOTA) performance on four event-based action recognition datasets using only 16 timesteps, surpassing other SNN methods. Moreover, it also achieves SOTA performance across all methods on three datasets, utilizing approximately 0.3\% of the parameters and 0.5\% of power consumption employed by artificial neural networks (ANNs). These results emphasize the significance of Point Cloud and pave the way for many ultra-low-power event-based data processing applications.

Motivation & Objective

• To address the inefficiency and sparsity loss in existing SNNs that convert asynchronous event data into static frames for processing.
• To develop a lightweight, end-to-end SNN architecture that natively processes sparse 3D event point clouds while preserving fine-grained temporal information.
• To achieve high accuracy with minimal parameters and ultra-low power consumption by leveraging surrogate training and identity mapping in a single-stage structure.
• To introduce a novel encoding scheme that preserves symmetry for negative relative position values in point cloud representations.
• To demonstrate the superiority of point cloud-based SNNs over frame-based and multi-stage SNN/ANN approaches in event-based action recognition.

Proposed method

• Representing event data as 3D space-time point clouds to retain sparsity and temporal resolution without frame conversion.
• Designing a singular-stage SNN architecture that jointly extracts local and global features using a dual-path structure with shared and separate branches.
• Employing a surrogate gradient method for back-propagation training, enabling end-to-end optimization without converting ANNs to SNNs.
• Introducing a novel encoding strategy that applies absolute values and centroid normalization to handle negative relative coordinates in point cloud input.
• Using identity mapping as the feature extractor to reduce parameter count and enhance training stability.
• Implementing a dual-branch fusion mechanism with element-wise addition (Add) to combine local and global features effectively.

Experimental results

Research questions

• RQ1 Can a point-based SNN architecture achieve superior action recognition performance on event camera data without converting events into frames?
• RQ2 How does a single-stage SNN design compare to multi-stage or residual-structured SNNs in terms of accuracy, parameter efficiency, and energy consumption?
• RQ3 What is the impact of different encoding strategies for relative position data—especially those involving negative values—on SNN performance?
• RQ4 To what extent can a lightweight SNN with minimal timesteps (e.g., 16) match or exceed the accuracy of larger ANNs and SNNs on event-based benchmarks?
• RQ5 Can a point cloud representation preserve temporal sparsity and improve energy efficiency in SNNs compared to voxelized or frame-based representations?

Key findings

• SpikePoint achieves state-of-the-art accuracy on five event-based action recognition datasets, including Daily DVS, DVS128 Gesture, and HMDB51-DVS, using only 16 timesteps.
• On the Daily DVS dataset, SpikePoint achieves 97.92% accuracy, surpassing previous SOTA SNNs and even some competitive ANNs.
• The model uses approximately 0.3% of the parameters and 0.5% of the power consumption of equivalent ANNs while maintaining comparable or better accuracy.
• The ablation study confirms that combining local and global feature extraction via a dual-path structure with Add fusion yields the highest performance, outperforming single-path or concatenation-based designs.
• The optimal timestep is 16, as increasing beyond this (e.g., 24 or 32) leads to accuracy degradation due to overfitting or temporal misalignment.
• The encoding strategy using absolute values and centroid normalization significantly improves performance, outperforming normalization and other variants, especially in handling negative coordinate values.