[Paper Review] Refined Temporal Pyramidal Compression-and-Amplification Transformer for 3D Human Pose Estimation

Hanbing Liu, Wangmeng Xiang+5 more|arXiv (Cornell University)|Sep 4, 2023

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes the Refined Temporal Pyramidal Compression-and-Amplification (RTPCA) transformer for 3D human pose estimation, integrating a Temporal Pyramidal Compression-and-Amplification (TPCA) module for intra-block multi-scale feature learning and a Cross-Layer Refinement (XLR) module for inter-block attention enhancement. The method achieves state-of-the-art accuracy with minimal computational cost on benchmarks like Human3.6M, HumanEva-I, and MPI-INF-3DHP.

ABSTRACT

Accurately estimating the 3D pose of humans in video sequences requires both accuracy and a well-structured architecture. With the success of transformers, we introduce the Refined Temporal Pyramidal Compression-and-Amplification (RTPCA) transformer. Exploiting the temporal dimension, RTPCA extends intra-block temporal modeling via its Temporal Pyramidal Compression-and-Amplification (TPCA) structure and refines inter-block feature interaction with a Cross-Layer Refinement (XLR) module. In particular, TPCA block exploits a temporal pyramid paradigm, reinforcing key and value representation capabilities and seamlessly extracting spatial semantics from motion sequences. We stitch these TPCA blocks with XLR that promotes rich semantic representation through continuous interaction of queries, keys, and values. This strategy embodies early-stage information with current flows, addressing typical deficits in detail and stability seen in other transformer-based methods. We demonstrate the effectiveness of RTPCA by achieving state-of-the-art results on Human3.6M, HumanEva-I, and MPI-INF-3DHP benchmarks with minimal computational overhead. The source code is available at https://github.com/hbing-l/RTPCA.

Motivation & Objective

• To address the limited exploitation of intra- and inter-block attention mechanisms in existing transformer-based 3D pose estimation methods.
• To enhance multi-scale temporal feature representation within individual transformer blocks using a pyramidal compression-and-amplification structure.
• To improve inter-block communication by enabling dynamic interaction between keys and values across adjacent layers.
• To develop a lightweight yet highly effective architecture that balances accuracy and inference efficiency.
• To validate the framework on standard benchmarks with comprehensive ablation and qualitative analysis.

Proposed method

• The Temporal Pyramidal Compression-and-Amplification (TPCA) module uses adaptive pooling and transpose convolution to compress and then amplify features across temporal levels, enriching intra-block key and value representations.
• The Cross-Layer Refinement (XLR) module fuses keys and values from earlier and later blocks via a learnable attention mechanism, enhancing inter-block information flow.
• The XLR module is implemented as a residual connection with constant query projection, enabling stable and effective cross-layer attention fusion.
• The TPCA and XLR modules are stacked in a transformer encoder architecture, with feature refinement integrated directly into the attention computation.
• The framework is trained end-to-end using standard 3D pose estimation loss (MPJPE) and evaluated on Human3.6M, HumanEva-I, and MPI-INF-3DHP datasets.
• Ablation studies compare different compression/amplification methods and fusion strategies, with the optimal configuration selected based on MPJPE and latency.

Experimental results

Research questions

• RQ1 Can a pyramidal compression-and-amplification mechanism improve multi-scale temporal feature learning within individual transformer blocks for 3D pose estimation?
• RQ2 Does cross-layer attention fusion between adjacent blocks enhance model performance and stability compared to standard self-attention?
• RQ3 How does the integration of intra-block and inter-block refinement affect the trade-off between accuracy and computational efficiency?
• RQ4 What is the optimal configuration of compression and amplification operations in the TPCA module for minimal MPJPE and latency?
• RQ5 How does the XLR module compare to feature-based fusion methods in terms of performance and parameter efficiency?

Key findings

• The RTPCA model achieves an MPJPE of 20.0mm on the Human3.6M dataset, outperforming existing SOTA methods in both accuracy and latency.
• The combination of adaptive pooling and transpose convolution in the TPCA module yields the best performance, reducing MPJPE to 20.0mm compared to 20.9mm with other configurations.
• The XLR module significantly improves model stability, evidenced by the lowest acceleration error (Accel) of 1.82 mm/s², indicating superior temporal consistency.
• The residual XLR with constant query projection achieved the best MPJPE of 19.5mm, outperforming both feature-based and other attention-based fusion strategies.
• The ablation study confirms that attention-based cross-layer fusion is more effective than feature-based fusion, which increases parameters and degrades performance.
• Qualitative results show that RTPCA produces more accurate and robust 3D pose reconstructions, especially in occlusion and complex motion scenarios.