[Paper Review] Highly Efficient 3D Human Pose Tracking from Events with Spiking Spatiotemporal Transformer

Shihao Zou, Yuxuan Mu+6 more|arXiv (Cornell University)|Mar 16, 2023

Advanced Memory and Neural ComputingEngineering15 citations

TL;DR

The paper presents an end-to-end sparse deep learning approach using Spiking Neural Networks to track 3D human pose from event streams only, featuring a Spiking Spatiotemporal Transformer and a large synthetic SynEventHPD dataset, achieving substantial FLOPs reduction while surpassing state-of-the-art methods.

ABSTRACT

Event camera, as an asynchronous vision sensor capturing scene dynamics, presents new opportunities for highly efficient 3D human pose tracking. Existing approaches typically adopt modern-day Artificial Neural Networks (ANNs), such as CNNs or Transformer, where sparse events are converted into dense images or paired with additional gray-scale images as input. Such practices, however, ignore the inherent sparsity of events, resulting in redundant computations, increased energy consumption, and potentially degraded performance. Motivated by these observations, we introduce the first sparse Spiking Neural Networks (SNNs) framework for 3D human pose tracking based solely on events. Our approach eliminates the need to convert sparse data to dense formats or incorporate additional images, thereby fully exploiting the innate sparsity of input events. Central to our framework is a novel Spiking Spatiotemporal Transformer, which enables bi-directional spatiotemporal fusion of spike pose features and provides a guaranteed similarity measurement between binary spike features in spiking attention. Moreover, we have constructed a large-scale synthetic dataset, SynEventHPD, that features a broad and diverse set of 3D human motions, as well as much longer hours of event streams. Empirical experiments demonstrate the superiority of our approach over existing state-of-the-art (SOTA) ANN-based methods, requiring only 19.1% FLOPs and 3.6% energy cost. Furthermore, our approach outperforms existing SNN-based benchmarks in this task, highlighting the effectiveness of our proposed SNN framework. The dataset will be released upon acceptance, and code can be found at https://github.com/JimmyZou/HumanPoseTracking_SNN.

Motivation & Objective

• Tackle 3D human pose tracking using only event camera data without gray-scale frames.
• Develop an end-to-end SNN architecture leveraging a Spiking Spatiotemporal Transformer for bidirectional temporal fusion.
• Introduce a large-scale synthetic event-based dataset SynEventHPD to support diverse motions.
• Demonstrate superior performance and substantial computational efficiency over SOTA ANN and SNN baselines.

Proposed method

• Preprocess event streams into a sequence of event voxel grids to preserve temporal information.
• Use SEW-ResNet as the SNN backbone to extract pose spike features from voxel grids.
• Introduce a Spiking Spatiotemporal Transformer with bi-directional attention for temporal fusion of spike features.
• Regress SMPL pose and shape parameters from 2D pooled spike features via three parallel linear layers.
• Train end-to-end with losses on pose, shape, 3D and 2D joints to output time-aligned 3D meshes.

Figure 1: Overview of our approach. An end-to-end sparse deep learning approach is proposed for 3D single-person pose tracking over time from event streams only. Specifically, our method incorporates SEW ResNet as its backbone, and a novel Spiking Spatiotemporal Transformer to address one-directiona

Experimental results

Research questions

• RQ1 Can 3D human pose tracking be performed end-to-end from events only, without gray-scale frames?
• RQ2 Does a spiking spatiotemporal attention mechanism enable effective bidirectional temporal fusion in SNNs for pose tracking?
• RQ3 How does an entirely SNN-based approach compare to ANN/SNN hybrids in accuracy and compute on event-based pose tracking?
• RQ4 What is the impact of a large synthetic event-based dataset on generalization for event-driven pose tracking?

Key findings

• The method achieves superior performance over state-of-the-art ANN and baseline SNN methods on event-based 3D pose tracking.
• The approach reduces FLOPs by about 80% compared to SOTA methods.
• The Spiking Spatiotemporal Transformer enables bidirectional information flow to improve early-time pose estimation.
• A large-scale synthetic dataset SynEventHPD is introduced, totaling 45.72 hours of event streams from multiple motion datasets.
• The pipeline outputs SMPL parameters (beta, theta) and global translation over time, forming 3D meshes for each time step.

Figure 2: Pipeline of our sparse deep learning approach . It contains four main sections: (i) Preprcessing in Sec. 4.1 converts a stream of events into a sequence of event voxel grids of the same temporal length. (ii) SEW-ResNet [ 24 ] , introduced in Sec. 4.2 , is used as backbone to extract pose s