[Paper Review] Compositional Human Pose Regression

Xiao Sun, Jiaxiang Shang+2 more|arXiv (Cornell University)|Apr 1, 2017

Human Pose and Action RecognitionComputer Science55 citations

TL;DR

The paper introduces a structure-aware regression method that represents pose with bones and uses a compositional loss to exploit skeleton structure, improving 3D and 2D pose estimation and enabling unified training with mixed 2D/3D data.

ABSTRACT

Regression based methods are not performing as well as detection based methods for human pose estimation. A central problem is that the structural information in the pose is not well exploited in the previous regression methods. In this work, we propose a structure-aware regression approach. It adopts a reparameterized pose representation using bones instead of joints. It exploits the joint connection structure to define a compositional loss function that encodes the long range interactions in the pose. It is simple, effective, and general for both 2D and 3D pose estimation in a unified setting. Comprehensive evaluation validates the effectiveness of our approach. It significantly advances the state-of-the-art on Human3.6M and is competitive with state-of-the-art results on MPII.

Motivation & Objective

• Motivate regression-based pose estimation to leverage pose structure alongside traditional detection methods.
• Propose bone-based pose representation to yield more stable and learnable features than joints.
• Introduce a compositional loss that enforces long-range consistency along skeleton paths.
• Demonstrate unified training for 2D and 3D pose regression, including mixed 2D/3D data.
• Evaluate on 3D (Human3.6M) and 2D (MPII) benchmarks and compare with state-of-the-art methods.

Proposed method

• Reparameterize pose from joints to bones, defined as B_k = J_parent(k) - J_k.
• Minimize a bone-based regression loss L(B) as in L(B) = sum_k || tilde(B_k) - tilde(B_k_gt) ||_1.
• Introduce a compositional long-range loss L(B, P) that enforces relative joint positions Delta J_u,v along bone paths, computed as a sum of bones along the path with a differentiable compositional layer.
• Consider multiple joint pair sets P (joint, bone, both, all) to evaluate the impact of exploiting skeleton structure.
• Support unified 2D/3D regression with outputs of size 2K or 3K, mixed training data, and shared loss decomposition into xy and z components.
• Use ResNet-50 as backbone, pre-trained on ImageNet, fine-tuned with SGD for 25 epochs, with standard data augmentation and two-GPU training.

Experimental results

Research questions

• RQ1 Can a bone-based representation and a compositional loss better capture the pose structure compared to direct joint regression?
• RQ2 Does exploiting long-range skeletal constraints improve 3D and 2D pose accuracy and geometric validity?
• RQ3 Is it beneficial to mix 2D and 3D data during training for unified pose regression?
• RQ4 How does the proposed method perform on 3D Human3.6M and 2D MPII benchmarks relative to state-of-the-art?
• RQ5 What metrics reveal geometric validity beyond traditional joint localization accuracy?

Key findings

• The bone-based representation yields more stable bone lengths and improved geometric constraint metrics compared to joint-based baselines.
• The compositional loss (especially when using all joint pairs) significantly improves joint and bone errors, bone stability, and legality of joint angles on Human3.6M.
• Adding MPII 2D data to training substantially reduces Joint Error (e.g., from 102.2 to 64.2 under Protocol 2).
• Ours (all) achieves state-of-the-art results among regression methods and competitive performance with detection-based methods on MPII.
• On Human3.6M, Ours (all) reduces Joint Error to 92.4 mm and PA Joint Error to 67.5 mm under the reported setting, with further improvements when using mixed 2D+3D data.
• The approach yields plausible 3D poses for in-the-wild MPII images, indicating transfer learning potential.