[Paper Review] Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approach

Xingyi Zhou, Qixing Huang+3 more|arXiv (Cornell University)|Apr 8, 2017

Human Pose and Action RecognitionComputer Science29 references69 citations

TL;DR

Proposes an end-to-end weakly supervised framework that transfers 3D pose knowledge from indoor 3D-labeled data to in-the-wild 2D-labeled data, integrating 2D pose and depth regression with a geometric constraint to regularize 3D predictions.

ABSTRACT

In this paper, we study the task of 3D human pose estimation in the wild. This task is challenging due to lack of training data, as existing datasets are either in the wild images with 2D pose or in the lab images with 3D pose. We propose a weakly-supervised transfer learning method that uses mixed 2D and 3D labels in a unified deep neutral network that presents two-stage cascaded structure. Our network augments a state-of-the-art 2D pose estimation sub-network with a 3D depth regression sub-network. Unlike previous two stage approaches that train the two sub-networks sequentially and separately, our training is end-to-end and fully exploits the correlation between the 2D pose and depth estimation sub-tasks. The deep features are better learnt through shared representations. In doing so, the 3D pose labels in controlled lab environments are transferred to in the wild images. In addition, we introduce a 3D geometric constraint to regularize the 3D pose prediction, which is effective in the absence of ground truth depth labels. Our method achieves competitive results on both 2D and 3D benchmarks.

Motivation & Objective

• Motivate 3D pose estimation in the wild despite limited 3D in-the-wild data.
• Propose an end-to-end network that jointly learns 2D pose and depth regression using both 2D and 3D data.
• Leverage intermediate 2D image features to improve 3D depth prediction.
• Introduce a 3D geometric constraint to regularize depth when only 2D labels are available.
• Demonstrate state-of-the-art or competitive performance on 3D and 2D benchmarks and show transfer to in-the-wild scenarios.

Proposed method

• A two-stage cascaded network with a 2D pose estimation module and a depth regression module.
• The 2D module uses the stacked hourglass architecture to produce 2D joint heatmaps.
• The depth module ingests intermediate feature maps from the 2D module along with 2D heatmaps to regress per-joint depth values.
• Training uses both 3D-labeled indoor data and 2D-labeled in-the-wild data in a unified end-to-end framework.
• A 3D geometric constraint enforces stability of bone-length ratios across joints when 3D depth labels are unavailable, implemented as a differentiable loss L_geo based on bone-length ratios within predefined bone groups.
• The overall loss combines L_2D and L_dep, with L_dep using Euclidean loss for 3D data and the geometric loss for 2D data, and a three-stage training scheme (Stage 1: 2D pretraining, Stage 2: 3D depth training with 2D fine-tuning, Stage 3: joint fine-tuning with geo-constraint).

Experimental results

Research questions

• RQ1 Can a unified, end-to-end network effectively learn 3D pose estimation in-the-wild by leveraging indoor 3D data and in-the-wild 2D data?
• RQ2 Does incorporating intermediate 2D features and a geometric depth-regularization loss improve 3D pose accuracy and pose plausibility in the wild?
• RQ3 How does the proposed weakly-supervised approach compare to fully supervised and other in-the-wild methods on standard 3D pose benchmarks?
• RQ4 Can a three-stage training protocol stabilize end-to-end training and maximize cross-domain transfer of depth information?

Key findings

• The proposed 3D+2D/w geo model significantly improves MPJPE on Human3.6M compared to baselines (e.g., 64.90 mm vs 82.44 mm without geo constraint).
• Joint training with both 2D and 3D data yields substantial gains over using 3D data alone or 2D data alone, demonstrating effective cross-domain transfer.
• The geometric constraint improves left-right bone symmetry and overall geometric validity, even on in-the-wild MPII validation and MPI-INF-3DHP datasets.
• 2D pose accuracy (PCKh@0.5) remains competitive, showing that depth supervision benefits the depth module without sacrificing 2D performance.
• The method achieves competitive to state-of-the-art results on in-the-wild benchmarks (MPI-INF-3DHP) without using their full training data, illustrating strong transfer capabilities.