[Paper Review] SMPLR: Deep SMPL reverse for 3D human pose and shape recovery

Meysam Madadi, Hugo Bertiche+1 more|arXiv (Cornell University)|Dec 27, 2018

Human Pose and Action RecognitionComputer Science38 references19 citations

TL;DR

This paper proposes SMPLR, a deep learning framework that reverses the SMPL body model to recover 3D human pose and shape from a single RGB image by using 3D joint predictions as an intermediate representation. By framing SMPLR as an autoencoder with a CNN-based encoder and SMPL as the decoder, the method avoids complex regularization and achieves state-of-the-art performance, reducing 3D joint error by 25mm on Human3.6M and 3.5mm on SURREAL compared to prior SMPL-based methods.

ABSTRACT

Current state-of-the-art in 3D human pose and shape recovery relies on deep neural networks and statistical morphable body models, such as the Skinned Multi-Person Linear model (SMPL). However, regardless of the advantages of having both body pose and shape, SMPL-based solutions have shown difficulties to predict 3D bodies accurately. This is mainly due to the unconstrained nature of SMPL, which may generate unrealistic body meshes. Because of this, regression of SMPL parameters is a difficult task, often addressed with complex regularization terms. In this paper we propose to embed SMPL within a deep model to accurately estimate 3D pose and shape from a still RGB image. We use CNN-based 3D joint predictions as an intermediate representation to regress SMPL pose and shape parameters. Later, 3D joints are reconstructed again in the SMPL output. This module can be seen as an autoencoder where the encoder is a deep neural network and the decoder is SMPL model. We refer to this as SMPL reverse (SMPLR). By implementing SMPLR as an encoder-decoder we avoid the need of complex constraints on pose and shape. Furthermore, given that in-the-wild datasets usually lack accurate 3D annotations, it is desirable to lift 2D joints to 3D without pairing 3D annotations with RGB images. Therefore, we also propose a denoising autoencoder (DAE) module between CNN and SMPLR, able to lift 2D joints to 3D and partially recover from structured error. We evaluate our method on SURREAL and Human3.6M datasets, showing improvement over SMPL-based state-of-the-art alternatives by about 4 and 25 millimeters, respectively.

Motivation & Objective

• To address the challenge of accurate 3D human pose and shape recovery from single RGB images, especially when SMPL-based methods produce unrealistic body meshes due to the model's many-to-one mapping and sensitivity to noise.
• To eliminate the need for complex regularization terms in SMPL parameter regression by using 3D joint predictions as an intermediate, disentangled representation.
• To improve robustness to structured errors in 2D-to-3D joint lifting, particularly from occlusions and depth ambiguity, in in-the-wild scenarios.
• To enable end-to-end training of a deep network that backpropagates SMPL-related errors through the CNN, improving joint estimation and mesh reconstruction.
• To achieve cross-dataset generalization by training SMPLR independently on synthetic mocap-like data, even with limited real-world 3D annotations.

Proposed method

• The method uses a volumetric stacked hourglass network (SHN) to predict 3D joint locations and sparse 3D landmarks from an RGB image as an intermediate representation.
• A denoising autoencoder (DAE) is introduced between the SHN and SMPLR to refine 2D or noisy 3D joint predictions by enforcing symmetry, bone length proportions, and human-consistent geometry.
• The DAE output is fed into a two-branch Multi-Layer Perceptron (MLP) network that regresses SMPL pose and shape parameters from the refined 3D joint and landmark data.
• The SMPL model acts as a differentiable decoder to generate a full 3D body mesh from the predicted SMPL parameters, enabling end-to-end training.
• The entire pipeline—SHN + DAE + MLP + SMPL—is trained end-to-end, allowing backpropagation of mesh and joint errors through the network.
• The framework is trained on synthetic mocap data for SMPLR, enabling generalization to real-world datasets like SURREAL and Human3.6M without requiring ground-truth SMPL parameters in the training data.

Experimental results

Research questions

• RQ1 Can a deep autoencoder-like structure that reverses the SMPL model improve 3D human body reconstruction from single RGB images?
• RQ2 Does using 3D joint predictions as an intermediate representation reduce the risk of generating unrealistic body meshes compared to direct SMPL parameter regression?
• RQ3 Can a denoising autoencoder effectively recover from structured errors in 2D-to-3D joint lifting, especially under occlusions or depth ambiguity?
• RQ4 To what extent can SMPLR generalize across datasets when trained on synthetic mocap data and fine-tuned on real images?
• RQ5 Does end-to-end training with differentiable SMPL rendering improve 3D joint and mesh accuracy compared to non-differentiable or weakly supervised alternatives?

Key findings

• On the Human3.6M dataset, SMPLR improves state-of-the-art SMPL-based methods by 25mm in 3D joint error under protocol 1, demonstrating significant gains in accuracy.
• On the SURREAL dataset, SMPLR reduces SMPL surface error by 3.5mm compared to the prior state-of-the-art, even without multi-task learning.
• The DAE module effectively removes structured errors in 2D-to-3D joint lifting, improving consistency in bone lengths and symmetry of predicted poses.
• The method achieves a silhouette Intersection over Union (IoU) of over 0.7 on all evaluated datasets without explicit training for this metric, indicating high-quality mesh rendering.
• The final end-to-end trained model ({SHN}_{e2e}^{final}) achieves state-of-the-art 3D joint estimation error of 40.8mm on SURREAL, matching the best-performing method without multi-task learning.
• Inference runs at 3 FPS on a GTX 1080Ti, with minimal performance degradation when using estimated image cropping instead of ground-truth, showing robustness to real-world preprocessing errors.