[論文レビュー] Highly Efficient 3D Human Pose Tracking from Events with Spiking Spatiotemporal Transformer

Shihao Zou, Yuxuan Mu+6 more|arXiv (Cornell University)|Mar 16, 2023

Advanced Memory and Neural Computing被引用数 15

ひとこと要約

この論文は、Spiking Neural Networksを用いたエンドツーエンドのスパース深層学習アプローチを提案し、イベントストリームのみから3D人間ポーズを追跡する。Spiking Spatiotemporal Transformerと大規模な合成SynEventHPDデータセットを特徴とし、最先端手法を凌駕しつつFLOPsを大幅に削減する。

ABSTRACT

Event camera, as an asynchronous vision sensor capturing scene dynamics, presents new opportunities for highly efficient 3D human pose tracking. Existing approaches typically adopt modern-day Artificial Neural Networks (ANNs), such as CNNs or Transformer, where sparse events are converted into dense images or paired with additional gray-scale images as input. Such practices, however, ignore the inherent sparsity of events, resulting in redundant computations, increased energy consumption, and potentially degraded performance. Motivated by these observations, we introduce the first sparse Spiking Neural Networks (SNNs) framework for 3D human pose tracking based solely on events. Our approach eliminates the need to convert sparse data to dense formats or incorporate additional images, thereby fully exploiting the innate sparsity of input events. Central to our framework is a novel Spiking Spatiotemporal Transformer, which enables bi-directional spatiotemporal fusion of spike pose features and provides a guaranteed similarity measurement between binary spike features in spiking attention. Moreover, we have constructed a large-scale synthetic dataset, SynEventHPD, that features a broad and diverse set of 3D human motions, as well as much longer hours of event streams. Empirical experiments demonstrate the superiority of our approach over existing state-of-the-art (SOTA) ANN-based methods, requiring only 19.1% FLOPs and 3.6% energy cost. Furthermore, our approach outperforms existing SNN-based benchmarks in this task, highlighting the effectiveness of our proposed SNN framework. The dataset will be released upon acceptance, and code can be found at https://github.com/JimmyZou/HumanPoseTracking_SNN.

研究の動機と目的

• 灰度フレームなしでイベントカメラデータのみを用いた3D人間ポーズ追跡を扱う。
• Spiking Spatiotemporal Transformerを活用した双方向の時間的融合を実現するエンドツーエンドのSNNアーキテクチャを開発。
• 大規模な合成イベントベースデータセット SynEventHPD を導入し、多様なモーションをサポート。
• SOTAのANNおよびSNNベースラインに対して、優れた性能と計算効率を示す。

提案手法

• イベントストリームをイベントボクセルグリッドのシーケンスに前処理して、時間情報を保持する。
• SNNバックボーンとしてSEW-ResNetを用い、ボクセルグリッドからポーズスパイク特徴を抽出する。
• スパイキングスパイティアルトランスフォーマーを導入し、双方向のアテンションによるスパイク特徴の時間的フュージョンを行う。
• 2Dプールされたスパイク特徴から、3つの並列線形層を介してSMPLポーズと形状パラメータを回帰する。
• ポーズ、形状、3Dおよび2Dジョイントの損失を用いてエンドツーエンドで訓練し、時刻合わせされた3Dメッシュを出力する。

Figure 1: Overview of our approach. An end-to-end sparse deep learning approach is proposed for 3D single-person pose tracking over time from event streams only. Specifically, our method incorporates SEW ResNet as its backbone, and a novel Spiking Spatiotemporal Transformer to address one-directiona

実験結果

リサーチクエスチョン

• RQ1 Can 3D human pose tracking be performed end-to-end from events only, without gray-scale frames?
• RQ2 Does a spiking spatiotemporal attention mechanism enable effective bidirectional temporal fusion in SNNs for pose tracking?
• RQ3 How does an entirely SNN-based approach compare to ANN/SNN hybrids in accuracy and compute on event-based pose tracking?
• RQ4 What is the impact of a large synthetic event-based dataset on generalization for event-driven pose tracking?

主な発見

• The method achieves superior performance over state-of-the-art ANN and baseline SNN methods on event-based 3D pose tracking.
• The approach reduces FLOPs by about 80% compared to SOTA methods.
• The Spiking Spatiotemporal Transformer enables bidirectional information flow to improve early-time pose estimation.
• A large-scale synthetic dataset SynEventHPD is introduced, totaling 45.72 hours of event streams from multiple motion datasets.
• The pipeline outputs SMPL parameters (beta, theta) and global translation over time, forming 3D meshes for each time step.

Figure 2: Pipeline of our sparse deep learning approach . It contains four main sections: (i) Preprcessing in Sec. 4.1 converts a stream of events into a sequence of event voxel grids of the same temporal length. (ii) SEW-ResNet [ 24 ] , introduced in Sec. 4.2 , is used as backbone to extract pose s