[Paper Review] SequentialPointNet: A strong parallelized point cloud sequence network for 3D action recognition

Xing Li, Qian Huang+3 more|arXiv (Cornell University)|Nov 16, 2021

Human Pose and Action Recognition71 references4 citations

TL;DR

SequentialPointNet is a parallelized 3D action recognition network that decouples intra-frame appearance and inter-frame motion encoding to efficiently capture spatiotemporal dynamics in point cloud sequences. By avoiding expensive spatio-temporal local neighborhood construction and using hierarchical pooling with temporal position encoding, it achieves state-of-the-art performance on three public datasets.

ABSTRACT

Point cloud sequences of 3D human actions exhibit unordered intra-frame spatial information and ordered interframe temporal information. In order to capture the spatiotemporal structures of the point cloud sequences, cross-frame spatio-temporal local neighborhoods around the centroids are usually constructed. However, the computationally expensive construction procedure of spatio-temporal local neighborhoods severely limits the parallelism of models. Moreover, it is unreasonable to treat spatial and temporal information equally in spatio-temporal local learning, because human actions are complicated along the spatial dimensions and simple along the temporal dimension. In this paper, to avoid spatio-temporal local encoding, we propose a strong parallelized point cloud sequence network referred to as SequentialPointNet for 3D action recognition. SequentialPointNet is composed of two serial modules, i.e., an intra-frame appearance encoding module and an inter-frame motion encoding module. For modeling the strong spatial structures of human actions, each point cloud frame is processed in parallel in the intra-frame appearance encoding module and the feature vector of each frame is output to form a feature vector sequence that characterizes static appearance changes along the temporal dimension. For modeling the weak temporal changes of human actions, in the inter-frame motion encoding module, the temporal position encoding and the hierarchical pyramid pooling strategy are implemented on the feature vector sequence. In addition, in order to better explore spatio-temporal content, multiple level features of human movements are aggregated before performing the end-to-end 3D action recognition. Extensive experiments conducted on three public datasets show that SequentialPointNet outperforms stateof-the-art approaches.

Motivation & Objective

• Address the computational bottleneck of spatio-temporal local neighborhood construction in 3D action recognition models.
• Decouple spatial appearance modeling from temporal motion modeling to better reflect the inherent structural differences in human actions.
• Enable strong parallelism in processing point cloud sequences by avoiding sequential neighborhood computation.
• Improve 3D action recognition accuracy by aggregating multi-level spatiotemporal features through hierarchical pooling.
• Develop a scalable and efficient architecture that maintains high performance without relying on complex local neighborhood learning.

Proposed method

• Process each point cloud frame independently in the intra-frame appearance encoding module to extract static spatial features in parallel.
• Generate a sequence of frame-level feature vectors that represent appearance changes over time.
• Apply temporal position encoding to the feature vector sequence to preserve temporal order and relative timing.
• Use a hierarchical pyramid pooling strategy in the inter-frame motion encoding module to capture long-range temporal dependencies efficiently.
• Aggregate multi-level features from different stages of the network to enrich spatiotemporal representation before final classification.
• Train the entire network end-to-end for 3D action recognition using standard cross-entropy loss.

Experimental results

Research questions

• RQ1 Can a parallelized architecture outperform existing spatio-temporal local neighborhood-based models in 3D action recognition while reducing computational cost?
• RQ2 How does decoupling spatial appearance and temporal motion modeling improve performance on point cloud sequence data?
• RQ3 To what extent can temporal position encoding and hierarchical pooling replace explicit spatio-temporal neighborhood construction?
• RQ4 Does multi-level feature aggregation enhance recognition accuracy on point cloud sequence benchmarks?
• RQ5 Can the proposed method achieve state-of-the-art performance without relying on complex local neighborhood computation?

Key findings

• SequentialPointNet achieves state-of-the-art performance on three public 3D action recognition benchmarks without using spatio-temporal local neighborhoods.
• The model demonstrates strong parallelism due to the avoidance of computationally expensive neighborhood construction, enabling efficient training and inference.
• The hierarchical pyramid pooling strategy effectively captures long-range temporal dependencies in the feature sequence.
• Temporal position encoding improves modeling of temporal order, contributing to better recognition accuracy.
• Multi-level feature aggregation leads to more robust and discriminative spatiotemporal representations.
• The ablation study confirms that the decoupled design of appearance and motion modules improves performance over joint spatio-temporal learning.