[Paper Review] MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video

Jinlu Zhang, Zhigang Tu+3 more|arXiv (Cornell University)|Mar 2, 2022

Human Pose and Action Recognition13 citations

TL;DR

MixSTE proposes a novel seq2seq transformer-based architecture for 3D human pose estimation that separately models temporal motion per joint using a joint-separation strategy and alternates between spatial and temporal transformer blocks to enhance spatio-temporal coherence. It achieves state-of-the-art performance, reducing MPJPE by 7.6% and P-MPJPE by 10.9% on benchmark datasets compared to prior methods.

ABSTRACT

Recent transformer-based solutions have been introduced to estimate 3D human pose from 2D keypoint sequence by considering body joints among all frames globally to learn spatio-temporal correlation. We observe that the motions of different joints differ significantly. However, the previous methods cannot efficiently model the solid inter-frame correspondence of each joint, leading to insufficient learning of spatial-temporal correlation. We propose MixSTE (Mixed Spatio-Temporal Encoder), which has a temporal transformer block to separately model the temporal motion of each joint and a spatial transformer block to learn inter-joint spatial correlation. These two blocks are utilized alternately to obtain better spatio-temporal feature encoding. In addition, the network output is extended from the central frame to entire frames of the input video, thereby improving the coherence between the input and output sequences. Extensive experiments are conducted on three benchmarks (Human3.6M, MPI-INF-3DHP, and HumanEva). The results show that our model outperforms the state-of-the-art approach by 10.9% P-MPJPE and 7.6% MPJPE. The code is available at https://github.com/JinluZhang1126/MixSTE.

Motivation & Objective

• To address the limitation of existing methods in modeling distinct motion trajectories of individual body joints across frames.
• To improve sequence coherence in 3D pose estimation by extending the output from a single central frame to the full input sequence.
• To enhance inference efficiency and reduce redundancy by adopting a seq2seq learning paradigm instead of repeated inference on overlapping sequences.
• To enable flexible input sequence lengths while maintaining high accuracy and temporal smoothness in long sequences.
• To provide a new, efficient, and accurate baseline for 3D human pose estimation using a mixed spatio-temporal transformer encoder.

Proposed method

• The model uses joint separation to treat each 2D keypoint as an individual token, enabling per-joint temporal motion learning and reducing feature dimensionality in the temporal domain.
• It employs an alternating design between spatial and temporal transformer blocks to progressively build global spatio-temporal representations.
• A mixed spatio-temporal encoder (MixSTE) integrates both spatial correlation among joints and temporal dynamics of individual joints through stacked transformer layers.
• The network is trained with a multi-loss objective: weighted MPJPE loss for accuracy, temporal consistency loss (TCLoss), and MPJVE loss for smoothness.
• The architecture supports arbitrary-length input sequences and outputs full 3D pose sequences, improving temporal coherence and reducing redundant computation.
• The method is end-to-end trainable and leverages self-attention mechanisms to model long-range dependencies across frames and joints.

Experimental results

Research questions

• RQ1 Can modeling individual joint motion trajectories improve 3D pose estimation accuracy in long video sequences?
• RQ2 How does a seq2seq transformer architecture compare to seq2frame approaches in terms of accuracy, smoothness, and inference efficiency?
• RQ3 To what extent can alternating spatial and temporal attention blocks enhance spatio-temporal feature learning in 3D pose estimation?
• RQ4 What loss functions are most effective for balancing accuracy and temporal smoothness in 3D pose sequence prediction?
• RQ5 Can a mixed spatio-temporal transformer encoder generalize across diverse input sequence lengths without performance degradation?

Key findings

• MixSTE achieves a 40.9 mm MPJPE on Human3.6M under Protocol 1, representing a 7.6% improvement over the previous state-of-the-art method.
• The model reduces P-MPJPE by 10.9% compared to the prior SOTA, demonstrating superior accuracy on long sequences.
• With a 243-frame input sequence, MixSTE achieves 40.9 mm MPJPE and 2.3 mm MPJVE, indicating strong temporal smoothness.
• Ablation studies confirm that combining WMPJPE loss, TCLoss, and MPJVE loss yields the best performance, with MPJVE decreasing by 2.3 mm.
• The model maintains high efficiency, achieving faster inference than seq2frame baselines due to full-sequence output and reduced redundancy.
• Hyperparameter ablation shows that a depth of 8, model dimension of 512, and input length of 243 yield optimal performance with minimal parameter growth.