[Paper Review] 3D Human Pose Estimation with Spatial and Temporal Transformers

Ce Zheng, Sijie Zhu+4 more|arXiv (Cornell University)|Mar 18, 2021

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes PoseFormer, a pure transformer-based model for 3D human pose estimation from 2D video sequences, using separate spatial and temporal transformer modules to model joint relationships within frames and long-range temporal dependencies across frames. It achieves state-of-the-art performance on Human3.6M and MPI-INF-3DHP with improved accuracy and generalization, especially when fine-tuned on smaller datasets.

ABSTRACT

Transformer architectures have become the model of choice in natural language processing and are now being introduced into computer vision tasks such as image classification, object detection, and semantic segmentation. However, in the field of human pose estimation, convolutional architectures still remain dominant. In this work, we present PoseFormer, a purely transformer-based approach for 3D human pose estimation in videos without convolutional architectures involved. Inspired by recent developments in vision transformers, we design a spatial-temporal transformer structure to comprehensively model the human joint relations within each frame as well as the temporal correlations across frames, then output an accurate 3D human pose of the center frame. We quantitatively and qualitatively evaluate our method on two popular and standard benchmark datasets: Human3.6M and MPI-INF-3DHP. Extensive experiments show that PoseFormer achieves state-of-the-art performance on both datasets. Code is available at \url{https://github.com/zczcwh/PoseFormer}

Motivation & Objective

• To address the limitations of convolutional networks in modeling long-range temporal dependencies and spatial joint relationships in 3D human pose estimation.
• To explore the feasibility and effectiveness of a pure transformer architecture for 2D-to-3D lifting in video-based 3D pose estimation.
• To design a model that maintains low parameter growth and computational cost even with long input sequences.
• To evaluate the model’s performance and generalization ability on standard benchmarks, including small-scale datasets.
• To analyze attention maps to understand how the model captures spatial and temporal dependencies.

Proposed method

• PoseFormer uses a spatial transformer module that processes 2D joint coordinates per frame, modeling local relationships between joints via multi-head self-attention with learnable positional embeddings.
• The spatial transformer outputs a latent feature representation per frame, preserving spatial structure and joint relationships.
• A temporal transformer module then processes these spatial representations across frames, capturing long-range temporal dependencies through self-attention over the sequence of frame-level features.
• The model treats each frame’s 2D pose as a sequence of 17 joint tokens, with positional embeddings applied to both spatial and temporal dimensions to preserve spatial and temporal order.
• The final 3D pose is predicted for the center frame using a learnable query vector that attends to the output of the temporal transformer.
• The model is trained end-to-end using standard regression loss on 3D joint coordinates, with no convolutional layers involved.

Experimental results

Research questions

• RQ1 Can a pure transformer architecture outperform convolutional baselines in 3D human pose estimation from 2D video sequences?
• RQ2 How effectively can a spatial-temporal transformer design model local joint relationships and long-range temporal dependencies simultaneously?
• RQ3 Does the proposed architecture generalize well to smaller datasets when pre-trained on large-scale data?
• RQ4 How do attention maps in the spatial and temporal transformers reflect meaningful human body structure and motion patterns?
• RQ5 What is the trade-off between model complexity, inference speed, and accuracy in long-sequence 3D pose estimation?

Key findings

• PoseFormer achieves state-of-the-art performance on the Human3.6M dataset, outperforming previous methods in terms of MPJPE (mean per-joint position error).
• On the MPI-INF-3DHP dataset, PoseFormer achieves the lowest MPJPE among all compared methods, demonstrating strong generalization and robustness.
• The model shows improved accuracy with longer input sequences, indicating effective modeling of long-range temporal dependencies.
• When fine-tuned on the smaller HumanEva dataset, PoseFormer achieves a significant performance boost, confirming strong generalization from large-scale pre-training.
• Attention visualization reveals that the spatial transformer learns meaningful joint groupings (e.g., left/right limbs), and the temporal transformer captures long-range motion patterns across distant frames.
• Despite not being the fastest in inference speed, PoseFormer's inference time remains acceptable, as it does not bottleneck the overall 3D HPE pipeline when combined with a 2D pose detector.