[Paper Review] TFPose: Direct Human Pose Estimation with Transformers

Weian Mao, Yongtao Ge+4 more|arXiv (Cornell University)|Mar 29, 2021

Human Pose and Action RecognitionComputer Science39 references57 citations

TL;DR

TFPose reframes 2D human pose estimation as a sequence prediction task and uses a transformer to directly regress keypoint coordinates, achieving state-of-the-art results among regression-based methods and competitive performance with heatmap-based methods.

ABSTRACT

We propose a human pose estimation framework that solves the task in the regression-based fashion. Unlike previous regression-based methods, which often fall behind those state-of-the-art methods, we formulate the pose estimation task into a sequence prediction problem that can effectively be solved by transformers. Our framework is simple and direct, bypassing the drawbacks of the heatmap-based pose estimation. Moreover, with the attention mechanism in transformers, our proposed framework is able to adaptively attend to the features most relevant to the target keypoints, which largely overcomes the feature misalignment issue of previous regression-based methods and considerably improves the performance. Importantly, our framework can inherently take advantages of the structured relationship between keypoints. Experiments on the MS-COCO and MPII datasets demonstrate that our method can significantly improve the state-of-the-art of regression-based pose estimation and perform comparably with the best heatmap-based pose estimation methods.

Motivation & Objective

• Address the regression-based pose estimation gap with heatmap-based methods.
• Leverage transformers to model spatial-temporal dependencies among keypoints.
• Eliminate heatmap post-processing and quantization limitations.
• Enable end-to-end differentiable training for pose estimation.

Proposed method

• Formulate single-person pose estimation as a K-length sequence prediction of keypoints.
• Use a CNN backbone to extract multi-level features then fuse with a transformer encoder.
• Deploy a transformer decoder with learnable keypoint queries to predict K 2D coordinates.
• Incorporate query-to-query attention to capture structured relationships among keypoints.
• Train with a combination of L1 regression loss across decoder layers and an auxiliary heatmap loss to accelerate convergence.
• Predict refinements Delta y_d in subsequent decoder layers for progressive improvement.

Experimental results

Research questions

• RQ1 Can a transformer-based regression framework surpass traditional regression-based pose methods?
• RQ2 Does incorporating query-to-query attention help model the structure among keypoints?
• RQ3 How does auxiliary heatmap supervision affect convergence and accuracy in a transformer-based pose model?
• RQ4 What is the trade-off between encoder/decoder configurations and pose estimation accuracy for COCO MPII?
• RQ5 How close can a regression-based method with transformers get to heatmap-based state-of-the-art on standard benchmarks?

Key findings

• TFPose significantly improves regression-based pose estimation, outperforming prior regression methods on COCO and MPII.
• Using query-to-query attention yields measurable AP gains (~1.3% AP on COCO val2017).
• A 256-channel Transformer encoder input outperforms 128-channel configurations in AP.
• Increasing decoder depth yields gains up to the third layer then saturates; best results reported with 6 decoders.
• An auxiliary heatmap loss accelerates convergence and boosts AP by about 2.3 percentage points.
• TFPose with ResNet-50 backbones achieves competitive results against heatmap-based methods with lower GFLOPs in COCO test-dev (e.g., 70.9–72.2 AP depending on config).