[Paper Review] TokenPose: Learning Keypoint Tokens for Human Pose Estimation

Yanjie Li, Shoukui Zhang+5 more|arXiv (Cornell University)|Apr 8, 2021

Human Pose and Action Recognition45 references18 citations

TL;DR

TokenPose introduces a novel Transformer-based framework that represents human keypoints as learnable tokens to explicitly model both visual appearance and anatomical constraint relationships. By jointly attending to visual tokens and keypoint tokens, it achieves state-of-the-art accuracy on COCO with 80.6% fewer parameters and 75.3% fewer FLOPs than prior CNN-based models.

ABSTRACT

Human pose estimation deeply relies on visual clues and anatomical constraints between parts to locate keypoints. Most existing CNN-based methods do well in visual representation, however, lacking in the ability to explicitly learn the constraint relationships between keypoints. In this paper, we propose a novel approach based on Token representation for human Pose estimation~(TokenPose). In detail, each keypoint is explicitly embedded as a token to simultaneously learn constraint relationships and appearance cues from images. Extensive experiments show that the small and large TokenPose models are on par with state-of-the-art CNN-based counterparts while being more lightweight. Specifically, our TokenPose-S and TokenPose-L achieve $72.5$ AP and $75.8$ AP on COCO validation dataset respectively, with significant reduction in parameters ($\downarrow80.6\%$; $\downarrow$ $56.8\%$) and GFLOPs ($\downarrow$ $75.3\%$; $\downarrow$ $24.7\%$). Code is publicly available.

Motivation & Objective

• To address the limitation of CNN-based pose estimators in explicitly modeling anatomical constraint relationships between keypoints.
• To unify visual appearance and structural constraint learning within a single, differentiable framework.
• To explore the feasibility of pure Transformer architectures for 2D human pose estimation without pre-training.
• To reduce model complexity and computational cost while maintaining or improving performance.

Proposed method

• Each keypoint is represented as a learnable, class-specific token that attends to visual tokens and other keypoint tokens via self- and cross-attention.
• Visual tokens are generated by dividing the input image into patches and projecting them into fixed-dimensional embeddings.
• The model uses a hybrid or pure Transformer encoder architecture to process interactions between visual and keypoint tokens.
• Keypoint tokens are initialized as random parameters and jointly optimized during training to encode statistical relationships between joints.
• Final keypoint locations are predicted by attending to visual tokens based on the final representation of keypoint tokens.
• The attention weights from the final self-attention layer are visualized to analyze how constraints (e.g., adjacency, symmetry) are learned.

Experimental results

Research questions

• RQ1 Can learnable keypoint tokens effectively model anatomical constraint relationships between joints in a self-supervised manner?
• RQ2 How does the interaction between keypoint tokens and visual tokens improve pose estimation accuracy?
• RQ3 Can a pure Transformer architecture without pre-training achieve competitive performance in 2D human pose estimation?
• RQ4 To what extent can the model reduce parameter count and FLOPs while maintaining SOTA accuracy?

Key findings

• TokenPose-S achieves 72.5 AP on COCO validation set with 80.6% fewer parameters and 75.3% fewer FLOPs than state-of-the-art CNN models.
• TokenPose-L achieves 75.8 AP on COCO, outperforming many larger CNN-based models while reducing parameters by 56.8% and FLOPs by 24.7%.
• The model learns meaningful prior knowledge: keypoint tokens show high similarity to adjacent and symmetric joints, as confirmed by attention scores and inner product matrices.
• Visualizations show that keypoint tokens progressively attend to relevant body parts (e.g., knee for ankle), and use symmetric joints as cues for occluded keypoint prediction.
• The pure Transformer-based TokenPose-T achieves competitive performance without pre-training, demonstrating the effectiveness of the token-based design.
• The attention mechanism reveals that constraint learning is implicitly encoded in keypoint tokens, with top-2 attention heads often corresponding to adjacent and symmetric joints.