[Paper Review] MPT: Mesh Pre-Training with Transformers for Human Pose and Mesh Reconstruction

Kevin Lin, Chung-Ching Lin+3 more|arXiv (Cornell University)|Nov 24, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes Mesh Pre-Training (MPT), a self-supervised pre-training method for 3D human pose and mesh reconstruction using large-scale MoCap data. By leveraging MoCap-generated heatmaps and a novel Masked Heatmap Modeling (MHM) objective, MPT enables strong zero-shot inference and state-of-the-art performance on Human3.6M, 3DPW, and FreiHAND without requiring paired image-mesh data during pre-training.

ABSTRACT

Traditional methods of reconstructing 3D human pose and mesh from single images rely on paired image-mesh datasets, which can be difficult and expensive to obtain. Due to this limitation, model scalability is constrained as well as reconstruction performance. Towards addressing the challenge, we introduce Mesh Pre-Training (MPT), an effective pre-training strategy that leverages large amounts of MoCap data to effectively perform pre-training at scale. We introduce the use of MoCap-generated heatmaps as input representations to the mesh regression transformer and propose a Masked Heatmap Modeling approach for improving pre-training performance. This study demonstrates that pre-training using the proposed MPT allows our models to perform effective inference without requiring fine-tuning. We further show that fine-tuning the pre-trained MPT model considerably improves the accuracy of human mesh reconstruction from single images. Experimental results show that MPT outperforms previous state-of-the-art methods on Human3.6M and 3DPW datasets. As a further application, we benchmark and study MPT on the task of 3D hand reconstruction, showing that our generic pre-training scheme generalizes well to hand pose estimation and achieves promising reconstruction performance.

Motivation & Objective

• To address the scarcity and cost of paired image-mesh datasets for 3D human mesh reconstruction.
• To improve model scalability and performance by leveraging large-scale MoCap data without requiring image supervision.
• To develop a generic pre-training framework that generalizes across body parts, including hands.
• To enable effective inference without fine-tuning by learning generalizable human pose and shape representations.

Proposed method

• Pre-train a mesh regression transformer using MoCap-generated heatmaps synthesized from 3D mesh data via virtual camera projections.
• Introduce Masked Heatmap Modeling (MHM), a masked autoencoding objective that improves self-supervised representation learning on heatmap inputs.
• Use off-the-shelf 2D pose models (e.g., HigherHRNet) to extract feature maps from input images during inference.
• Fine-tune the entire pipeline end-to-end using 2D-3D supervision to adapt pre-trained features to single-image reconstruction.
• Scale pre-training to 2 million human meshes to learn rich, generalizable human body priors.
• Apply the same pre-training scheme to 3D hand reconstruction using synthetic MoCap data (Complement) to demonstrate generalization.

Experimental results

Research questions

• RQ1 Can MoCap data be effectively leveraged for self-supervised pre-training in 3D human mesh reconstruction without paired images?
• RQ2 Does using MoCap-generated heatmaps as input representations improve pre-training performance compared to direct 3D joint regression?
• RQ3 Can the pre-trained model generalize to zero-shot inference on real images without fine-tuning?
• RQ4 How does the scale of MoCap data impact pre-training performance?
• RQ5 Can the proposed MPT framework generalize to other body parts, such as hands, with minimal adaptation?

Key findings

• MPT achieves state-of-the-art performance on Human3.6M with a PA-MPJPE of 5.4 mm and PA-MPJPE of 5.6 mm, outperforming prior methods.
• On the 3DPW dataset, MPT achieves a PA-MPJPE of 5.6 mm, demonstrating strong generalization to unconstrained images.
• On the FreiHAND dataset, MPT achieves a PA-MPJPE of 5.6 mm and F@15 mm of 0.988, surpassing previous SOTA methods.
• The ablation study shows that MHM improves PA-MPJPE by a large margin, confirming its effectiveness in self-supervised learning.
• Pre-training with increasing data scales improves zero-shot performance, with 100% data achieving 58.4 PA-MPJPE on Human3.6M without fine-tuning.
• The method generalizes well to hand reconstruction, achieving SOTA results on FreiHAND using only synthetic MoCap data for pre-training.