[论文解读] Highly Efficient 3D Human Pose Tracking from Events with Spiking Spatiotemporal Transformer

Shihao Zou, Yuxuan Mu+6 more|arXiv (Cornell University)|Mar 16, 2023

Advanced Memory and Neural Computing被引用 15

一句话总结

该论文提出一种端到端的稀疏深度学习方法，使用 Spiking Neural Networks 仅从事件流跟踪 3D 人体姿态，特色是一个 Spiking Spatiotemporal Transformer 和一个大型合成的 SynEventHPD 数据集，在显著降低 FLOPs 的同时超越最先进方法。

ABSTRACT

Event camera, as an asynchronous vision sensor capturing scene dynamics, presents new opportunities for highly efficient 3D human pose tracking. Existing approaches typically adopt modern-day Artificial Neural Networks (ANNs), such as CNNs or Transformer, where sparse events are converted into dense images or paired with additional gray-scale images as input. Such practices, however, ignore the inherent sparsity of events, resulting in redundant computations, increased energy consumption, and potentially degraded performance. Motivated by these observations, we introduce the first sparse Spiking Neural Networks (SNNs) framework for 3D human pose tracking based solely on events. Our approach eliminates the need to convert sparse data to dense formats or incorporate additional images, thereby fully exploiting the innate sparsity of input events. Central to our framework is a novel Spiking Spatiotemporal Transformer, which enables bi-directional spatiotemporal fusion of spike pose features and provides a guaranteed similarity measurement between binary spike features in spiking attention. Moreover, we have constructed a large-scale synthetic dataset, SynEventHPD, that features a broad and diverse set of 3D human motions, as well as much longer hours of event streams. Empirical experiments demonstrate the superiority of our approach over existing state-of-the-art (SOTA) ANN-based methods, requiring only 19.1% FLOPs and 3.6% energy cost. Furthermore, our approach outperforms existing SNN-based benchmarks in this task, highlighting the effectiveness of our proposed SNN framework. The dataset will be released upon acceptance, and code can be found at https://github.com/JimmyZou/HumanPoseTracking_SNN.

研究动机与目标

• Tackle 3D human pose tracking using only event camera data without gray-scale frames.
• Develop an end-to-end SNN architecture leveraging a Spiking Spatiotemporal Transformer for bidirectional temporal fusion.
• Introduce a large-scale synthetic event-based dataset SynEventHPD to support diverse motions.
• Demonstrate superior performance and substantial computational efficiency over SOTA ANN and SNN baselines.

提出的方法

• Preprocess event streams into a sequence of event voxel grids to preserve temporal information.
• Use SEW-ResNet as the SNN backbone to extract pose spike features from voxel grids.
• Introduce a Spiking Spatiotemporal Transformer with bi-directional attention for temporal fusion of spike features.
• Regress SMPL pose and shape parameters from 2D pooled spike features via three parallel linear layers.
• Train end-to-end with losses on pose, shape, 3D and 2D joints to output time-aligned 3D meshes.

Figure 1: Overview of our approach. An end-to-end sparse deep learning approach is proposed for 3D single-person pose tracking over time from event streams only. Specifically, our method incorporates SEW ResNet as its backbone, and a novel Spiking Spatiotemporal Transformer to address one-directiona

实验结果

研究问题

• RQ1 Can 3D human pose tracking be performed end-to-end from events only, without gray-scale frames?
• RQ2 Does a spiking spatiotemporal attention mechanism enable effective bidirectional temporal fusion in SNNs for pose tracking?
• RQ3 How does an entirely SNN-based approach compare to ANN/SNN hybrids in accuracy and compute on event-based pose tracking?
• RQ4 What is the impact of a large synthetic event-based dataset on generalization for event-driven pose tracking?

主要发现

• The method achieves superior performance over state-of-the-art ANN and baseline SNN methods on event-based 3D pose tracking.
• The approach reduces FLOPs by about 80% compared to SOTA methods.
• The Spiking Spatiotemporal Transformer enables bidirectional information flow to improve early-time pose estimation.
• A large-scale synthetic dataset SynEventHPD is introduced, totaling 45.72 hours of event streams from multiple motion datasets.
• The pipeline outputs SMPL parameters (beta, theta) and global translation over time, forming 3D meshes for each time step.

Figure 2: Pipeline of our sparse deep learning approach . It contains four main sections: (i) Preprcessing in Sec. 4.1 converts a stream of events into a sequence of event voxel grids of the same temporal length. (ii) SEW-ResNet [ 24 ] , introduced in Sec. 4.2 , is used as backbone to extract pose s