[Paper Review] PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences

Hehe Fan, Xin Yu+3 more|arXiv (Cornell University)|May 27, 2022

Human Pose and Action Recognition58 references69 citations

TL;DR

PSTNet introduces a point-based spatio-temporal convolution that decouples space and time for dynamic point clouds, forming a hierarchical network for 3D action recognition and 4D semantic segmentation.

ABSTRACT

Point cloud sequences are irregular and unordered in the spatial dimension while exhibiting regularities and order in the temporal dimension. Therefore, existing grid based convolutions for conventional video processing cannot be directly applied to spatio-temporal modeling of raw point cloud sequences. In this paper, we propose a point spatio-temporal (PST) convolution to achieve informative representations of point cloud sequences. The proposed PST convolution first disentangles space and time in point cloud sequences. Then, a spatial convolution is employed to capture the local structure of points in the 3D space, and a temporal convolution is used to model the dynamics of the spatial regions along the time dimension. Furthermore, we incorporate the proposed PST convolution into a deep network, namely PSTNet, to extract features of point cloud sequences in a hierarchical manner. Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of PSTNet to model point cloud sequences.

Motivation & Objective

• Motivate modeling of dynamic, irregular point clouds without voxelization or tracking.
• Propose PST convolution to decouple spatial structure from temporal dynamics in point sequences.
• Build PSTNet architectures for sequence-level classification and point-level prediction.
• Demonstrate effectiveness on 3D action recognition and 4D semantic segmentation benchmarks.

Proposed method

• Decouple space and time in point cloud sequences and define PST convolution.
• Spatial convolution over local 3D neighborhoods using a learned displacement-based kernel function f(delta; theta).
• Temporal convolution over a local sequence of frames to capture dynamics.
• Construct point tubes with temporal anchor frames and FPS-based spatial anchors to enable spatio-temporal convolution.
• Introduce PST transposed convolution to upsample and interpolate features for dense point-level predictions.
• Assemble PSTNet architectures with multiple PST layers (and transposed layers) for action recognition and semantic segmentation.

Experimental results

Research questions

• RQ1 Can decoupling spatial structure and temporal dynamics improve learning on dynamic point clouds?
• RQ2 Does PSTNet offer superior accuracy and efficiency for 3D action recognition and 4D semantic segmentation compared with prior methods?
• RQ3 How do temporal kernel size and spatial radius affect performance on point cloud sequence tasks?

Key findings

• PSTNet achieves state-of-the-art results on MSR-Action3D, surpassing previous methods across frame settings up to 24 frames.
• On NTU RGB+D 60/120, PSTNet shows strong improvements over skeleton-, depth-, and voxel-based baselines.
• For 4D semantic segmentation on Synthia 4D, PSTNet with temporal modeling (l=3) outperforms baselines and uses fewer parameters than some competitors.
• Ablation indicates longer clips and appropriate temporal kernel size improve action recognition, while spatial radius trades off local structure capture and discriminativeness.
• Visualization shows PSTNet activates more strongly at moving regions, indicating effective motion modeling.