[Paper Review] HEMlets PoSh: Learning Part-Centric Heatmap Triplets for 3D Human Pose and Shape Estimation

Kun Zhou, Xiaoguang Han+3 more|arXiv (Cornell University)|Mar 10, 2020

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes HEMlets PoSh, a novel method for 3D human pose and shape estimation from single RGB images using Part-Centric Heatmap Triplets (HEMlets) as an intermediate representation. HEMlets encode relative depth ordering and spatial co-location likelihoods for kinematically connected joint pairs, enabling a progressive learning strategy that improves 3D pose estimation accuracy by ~20% over state-of-the-art methods on Human3.6M, while generalizing well to in-the-wild images.

ABSTRACT

Estimating 3D human pose from a single image is a challenging task. This work attempts to address the uncertainty of lifting the detected 2D joints to the 3D space by introducing an intermediate state-Part-Centric Heatmap Triplets (HEMlets), which shortens the gap between the 2D observation and the 3D interpretation. The HEMlets utilize three joint-heatmaps to represent the relative depth information of the end-joints for each skeletal body part. In our approach, a Convolutional Network (ConvNet) is first trained to predict HEMlets from the input image, followed by a volumetric joint-heatmap regression. We leverage on the integral operation to extract the joint locations from the volumetric heatmaps, guaranteeing end-to-end learning. Despite the simplicity of the network design, the quantitative comparisons show a significant performance improvement over the best-of-grade methods (e.g. $20\%$ on Human3.6M). The proposed method naturally supports training with "in-the-wild" images, where only weakly-annotated relative depth information of skeletal joints is available. This further improves the generalization ability of our model, as validated by qualitative comparisons on outdoor images. Leveraging the strength of the HEMlets pose estimation, we further design and append a shallow yet effective network module to regress the SMPL parameters of the body pose and shape. We term the entire HEMlets-based human pose and shape recovery pipeline HEMlets PoSh. Extensive quantitative and qualitative experiments on the existing human body recovery benchmarks justify the state-of-the-art results obtained with our HEMlets PoSh approach.

Motivation & Objective

• To address the ambiguity and large domain gap in lifting 2D joints to 3D space by introducing an intermediate, learnable representation.
• To improve generalization to unconstrained, in-the-wild images by leveraging weakly-supervised relative depth annotations.
• To enable end-to-end, differentiable 3D joint regression via soft-argmax on volumetric heatmaps.
• To develop a simple yet effective pipeline for joint 3D pose and SMPL body shape regression from a single image.
• To demonstrate state-of-the-art performance on standard benchmarks including Human3.6M, 3DPW, SURREAL, and UP-3D.

Proposed method

• The method introduces Part-Centric Heatmap Triplets (HEMlets), which represent three relative depth states (forward, backward, or equal depth) for each skeletal part’s end-joints using three 2D heatmaps.
• A two-stage ConvNet first predicts 2D joint heatmaps and HEMlets from the input RGB image, then regresses 3D volumetric heatmaps using high-level features and HEMlets as auxiliary supervision.
• The 3D joint locations are extracted via differentiable soft-argmax operation, enabling end-to-end training with gradient backpropagation.
• A shallow, lightweight network head is appended to regress SMPL body shape (β) and pose (θ) parameters from the same feature space.
• The framework supports weakly-supervised pretraining using datasets with only relative depth ordering annotations, enhancing robustness to real-world images.
• The model is trained progressively: first 2D joint detection and HEMlets prediction, then 3D heatmap regression, reducing overfitting and improving convergence.

Experimental results

Research questions

• RQ1 Can an intermediate, part-centric representation that encodes relative depth ordering improve 3D human pose estimation from single images?
• RQ2 Does learning HEMlets as a bridge between 2D observations and 3D predictions reduce the domain gap and improve generalization to in-the-wild images?
• RQ3 To what extent does the accuracy of 3D pose estimation influence the quality of subsequent SMPL body shape and pose regression?
• RQ4 Can a simple, modular network design with progressive supervision achieve state-of-the-art performance in 3D human body recovery?
• RQ5 How effective is the proposed method when trained with only weakly-annotated relative depth information?

Key findings

• HEMlets PoSh achieves a mean Per-Joint Position Error (MPJPE) of 39.9mm on the Human3.6M benchmark, representing a ~20% improvement over the best competing method [42].
• The method generalizes well to in-the-wild images, as validated by qualitative results on outdoor scenes with challenging lighting and occlusions.
• On the 3DPW dataset, the full HEMlets PoSh model achieves a reconstruction error of 58.8mm, outperforming both ablated versions and prior methods.
• Ablation studies show that the 3D joint regression branch contributes more significantly to body mesh recovery than the high-level feature branch alone.
• Replacing estimated pose with ground-truth pose reduces error to 9.4mm, indicating that pose accuracy is the dominant factor in body shape recovery quality.
• The method achieves the lowest surface error on both SURREAL and UP-3D datasets, confirming state-of-the-art performance across multiple benchmarks.