[Paper Review] Benchmarking and Analyzing 3D Human Pose and Shape Estimation Beyond Algorithms

Hui En Pang, Zhongang Cai+3 more|arXiv (Cornell University)|Sep 21, 2022

Human Pose and Action Recognition8 citations

TL;DR

This paper presents a comprehensive benchmarking study of 3D human pose and shape estimation beyond algorithmic design, evaluating datasets, backbones, and training strategies. By optimizing data selection, using strong pre-trained backbones (e.g., Twins-SVT), and applying L1 loss with data augmentation, the authors achieve a PA-MPJPE of 47.3 mm on 3DPW—surpassing state-of-the-art models with simpler architectures, and establishing strong, fair baselines for future research.

ABSTRACT

3D human pose and shape estimation (a.k.a. "human mesh recovery") has achieved substantial progress. Researchers mainly focus on the development of novel algorithms, while less attention has been paid to other critical factors involved. This could lead to less optimal baselines, hindering the fair and faithful evaluations of newly designed methodologies. To address this problem, this work presents the first comprehensive benchmarking study from three under-explored perspectives beyond algorithms. 1) Datasets. An analysis on 31 datasets reveals the distinct impacts of data samples: datasets featuring critical attributes (i.e. diverse poses, shapes, camera characteristics, backbone features) are more effective. Strategical selection and combination of high-quality datasets can yield a significant boost to the model performance. 2) Backbones. Experiments with 10 backbones, ranging from CNNs to transformers, show the knowledge learnt from a proximity task is readily transferable to human mesh recovery. 3) Training strategies. Proper augmentation techniques and loss designs are crucial. With the above findings, we achieve a PA-MPJPE of 47.3 mm on the 3DPW test set with a relatively simple model. More importantly, we provide strong baselines for fair comparisons of algorithms, and recommendations for building effective training configurations in the future. Codebase is available at http://github.com/smplbody/hmr-benchmarks

Motivation & Objective

• To address the lack of systematic evaluation of non-algorithmic factors in 3D human mesh recovery, such as dataset choice, backbone selection, and training strategies.
• To identify how data diversity (poses, shapes, camera views, features) influences model performance and to recommend optimal dataset combinations.
• To demonstrate that pretraining on related tasks and using strong backbones (e.g., vision transformers) significantly improves performance.
• To establish fair, strong baselines using optimized configurations for HMR and other models, enabling reliable comparison of new algorithms.
• To provide a public codebase for reproducibility and future benchmarking of 3D human mesh recovery models.

Proposed method

• The study evaluates 31 diverse datasets, analyzing their impact on model generalization using feature distributions (via UMAP) and performance on 3DPW.
• It benchmarks 10 backbones, including CNNs (ResNet, HRNet) and vision transformers (Twins-SVT), with ablation on pretraining from pose estimation models.
• The authors compare various data augmentation techniques and loss functions, finding L1 loss more effective than mixed losses for reducing noise and improving generalization.
• They conduct extensive ablation studies across multiple algorithms (HMR, PARE, SPIN, GraphCMR, MeshGraphormer) with optimized configurations.
• The study uses feature space analysis (pose, shape, camera, backbone features) to assess distribution shifts between training datasets and the 3DPW test set.
• A public codebase is released to enable reproducibility and facilitate future benchmarking with the proposed optimal configurations.

Experimental results

Research questions

• RQ1 How do different datasets impact 3D human mesh recovery performance, and which attributes (e.g., pose diversity, shape variation, camera characteristics) make a dataset more effective?
• RQ2 To what extent can knowledge from a related task (e.g., 2D pose estimation) be transferred via pretraining to improve 3D mesh recovery?
• RQ3 What is the optimal combination of data augmentation and loss function design for minimizing recovery error?
• RQ4 Can a simple model achieve state-of-the-art performance when trained with an optimal configuration of datasets, backbone, and training strategy?
• RQ5 How do distribution shifts in pose, shape, camera, and backbone features between training data and 3DPW affect model generalization?

Key findings

• Using a combination of high-quality datasets—especially those with diverse poses, shapes, and camera views—leads to substantial performance gains, with MuCo-3DHP reducing distribution shift compared to MPI-INF-3DHP.
• Pretraining on a strong 2D pose estimation model significantly improves backbone quality, with Twins-SVT and HRNet-W32 achieving superior performance when initialized this way.
• L1 loss outperforms mixed losses in minimizing noise and improving generalization, particularly when combined with data augmentation.
• The best-performing model achieves a PA-MPJPE of 47.3 mm on the 3DPW test set using a simple HMR-based architecture with optimized data mix and training strategy.
• The study establishes strong, fair baselines for HMR and other algorithms across multiple datasets and backbones, with performance improvements of up to 10–15 mm in PA-MPJPE over prior works.
• Feature distribution analysis reveals that datasets like Instavariety, COCO, and AGORA have more diverse and transferable feature representations, reducing domain shift to 3DPW.