[Paper Review] Not All Parts Are Created Equal: 3D Pose Estimation by Modelling Bi-directional Dependencies of Body Parts

Jue Wang, Shaoli Huang+2 more|arXiv (Cornell University)|May 20, 2019

Human Pose and Action Recognition43 references14 citations

TL;DR

This paper proposes a novel 3D human pose estimation method that models bi-directional dependencies between body parts based on their degrees of freedom (DOF), treating high-DOF joints (e.g., elbows) as dependent on lower-DOF ones (e.g., torso) while allowing them to constrain the latter. It introduces a pose attribute prediction head to estimate relative joint positions (front/back/on-plane) with respect to the torso, significantly improving limb estimation accuracy and achieving state-of-the-art performance on Human3.6M and MPI-INF-3DHP benchmarks.

ABSTRACT

Not all the human body parts have the same~degree of freedom~(DOF) due to the physiological structure. For example, the limbs may move more flexibly and freely than the torso does. Most of the existing 3D pose estimation methods, despite the very promising results achieved, treat the body joints equally and consequently often lead to larger reconstruction errors on the limbs. In this paper, we propose a progressive approach that explicitly accounts for the distinct DOFs among the body parts. We model parts with higher DOFs like the elbows, as dependent components of the corresponding parts with lower DOFs like the torso, of which the 3D locations can be more reliably estimated. Meanwhile, the high-DOF parts may, in turn, impose a constraint on where the low-DOF ones lie. As a result, parts with different DOFs supervise one another, yielding physically constrained and plausible pose-estimation results. To further facilitate the prediction of the high-DOF parts, we introduce a pose-attribute estimation, where the relative location of a limb joint with respect to the torso, which has the least DOF of a human body, is explicitly estimated and further fed to the joint-estimation module. The proposed approach achieves very promising results, outperforming the state of the art on several benchmarks.

Motivation & Objective

• To address the limitation in existing 3D pose estimation methods that treat all body joints equally, despite their varying degrees of freedom (DOF).
• To reduce reconstruction errors on highly flexible limbs by explicitly modeling their dependency on more stable, low-DOF body parts like the torso.
• To improve 3D pose estimation accuracy by introducing a reliable, multi-class pose attribute that encodes relative joint position with respect to the torso.
• To enable robust generalization to in-the-wild images through unsupervised domain adaptation, even without 3D annotations or pose-attribute supervision.

Proposed method

• Categorizes body parts into three DOF levels: torso (lowest DOF), proximal limbs (e.g., shoulders, elbows, knees), and distal limbs (e.g., wrists, ankles).
• Models high-DOF joints as dependent on lower-DOF body parts in a progressive, bi-directional manner, where both directions supervise each other to improve physical plausibility.
• Introduces a multi-task network to jointly predict 2D joint locations and a three-class pose attribute (front, back, on-plane) for each limb joint, providing a strong spatial prior.
• Uses the predicted pose attributes as input to the 3D pose estimation network, replacing regression-based depth estimation to reduce error propagation.
• Employs a two-stage framework: first estimating 2D poses and pose attributes, then refining 3D poses using dependency modeling.
• Applies unsupervised domain adaptation to transfer the model to in-the-wild images without requiring 3D annotations or pose-attribute labels.

Experimental results

Research questions

• RQ1 Can modeling bi-directional dependencies between body parts of different DOFs improve 3D pose estimation accuracy, especially for high-DOF limbs?
• RQ2 Does predicting a discrete pose attribute (relative position to the torso) provide a more reliable and effective prior than regression-based depth estimation for 3D pose refinement?
• RQ3 How does the proposed method generalize to unseen domains, such as in-the-wild images without 3D annotations?
• RQ4 To what extent does the pose attribute component contribute to performance gains, particularly on challenging joints like wrists and ankles?
• RQ5 Can the method achieve state-of-the-art performance on standard benchmarks like Human3.6M and MPI-INF-3DHP without relying on domain-specific training data?

Key findings

• The proposed method outperforms previous state-of-the-art methods on the Human3.6M dataset, achieving a mean per-joint error of 52.6 mm under Protocol #1, a significant improvement over the baseline.
• On the MPI-INF-3DHP benchmark, the method achieves 71.9% 3DPCK and 35.8 AUC, surpassing prior methods and demonstrating strong cross-domain generalization.
• The ablation study confirms that the pose attribute component reduces error by up to 10.4 mm on the wrist joint and 6.6 mm on the ankle, highlighting its effectiveness.
• The bi-directional dependency modeling alone reduces error by 1.4 mm on average compared to the baseline, showing its direct contribution to performance.
• Even without 3D annotations or pose-attribute supervision, the method generalizes well to in-the-wild images via unsupervised domain adaptation, maintaining strong performance.