[Paper Review] 3D Human Pose Estimation from a Single Image via Distance Matrix Regression

Francesc Moreno-Noguer|RECERCAT (Consorci de Serveis Universitaris de Catalunya)|Nov 28, 2016

Human Pose and Action Recognition47 references14 citations

TL;DR

This paper proposes a novel 3D human pose estimation method from a single image by regressing from 2D to 3D Euclidean Distance Matrices (EDMs) instead of direct 3D coordinate regression. Using simple fully connected or fully convolutional networks trained on synthesized data, the approach improves robustness to 2D detection noise and occlusions, achieving state-of-the-art performance on Humaneva-I and Human3.6M, with strong generalization to unconstrained images in the LSP dataset.

ABSTRACT

This paper addresses the problem of 3D human pose estimation from a single image. We follow a standard two-step pipeline by first detecting the 2D position of the $N$ body joints, and then using these observations to infer 3D pose. For the first step, we use a recent CNN-based detector. For the second step, most existing approaches perform 2$N$-to-3$N$ regression of the Cartesian joint coordinates. We show that more precise pose estimates can be obtained by representing both the 2D and 3D human poses using $N imes N$ distance matrices, and formulating the problem as a 2D-to-3D distance matrix regression. For learning such a regressor we leverage on simple Neural Network architectures, which by construction, enforce positivity and symmetry of the predicted matrices. The approach has also the advantage to naturally handle missing observations and allowing to hypothesize the position of non-observed joints. Quantitative results on Humaneva and Human3.6M datasets demonstrate consistent performance gains over state-of-the-art. Qualitative evaluation on the images in-the-wild of the LSP dataset, using the regressor learned on Human3.6M, reveals very promising generalization results.

Motivation & Objective

• Address the ill-posed nature of 3D human pose estimation from single RGB images by leveraging structural priors from joint configurations.
• Overcome limitations of direct 2D-to-3D Cartesian coordinate regression, which is sensitive to noise and lacks structural invariance.
• Improve generalization to unconstrained, real-world images by learning from synthetic data and using distance matrices that encode pose geometry.
• Enable hypothesis of missing or occluded body parts by modeling pairwise joint relationships through distance matrices.
• Achieve high accuracy and robustness using shallow neural networks trained on synthetic data with controlled noise and occlusion.

Proposed method

• Represent 2D and 3D human poses as N×N Euclidean Distance Matrices (EDMs), encoding pairwise distances between N body joints.
• Formulate the 3D pose estimation task as a 2D-to-3D EDM regression problem, where the input is the 2D EDM and the output is the predicted 3D EDM.
• Train fully connected (FConn) and fully convolutional (FConv) neural networks to map 2D EDMs to 3D EDMs, with architecture depth chosen for small input size (N=14 joints).
• Enforce structural constraints (positivity, symmetry) in the network output by construction, since EDMs are inherently symmetric and positive semi-definite.
• Reconstruct 3D joint coordinates from the predicted 3D EDM using Multidimensional Scaling (MDS), a standard technique for EDM embedding.
• Synthesize training data with 2D detector noise and joint occlusions to improve robustness, leveraging the fact that EDMs depend only on joint positions.

Experimental results

Research questions

• RQ1 Can representing 2D and 3D poses as distance matrices improve 3D pose estimation accuracy compared to direct 2D-to-3D coordinate regression?
• RQ2 Does distance matrix regression enhance robustness to 2D detection noise and partial occlusions of body joints?
• RQ3 Can a model trained on controlled, lab-based datasets (e.g., Human3.6M) generalize effectively to unconstrained, real-world images (e.g., LSP)?
• RQ4 To what extent can the model hypothesize the 3D positions of occluded or unobserved joints using only the visible joint distances?
• RQ5 How do simple neural network architectures perform in 2D-to-3D EDM regression, especially when trained on synthetic data with noise and occlusion?

Key findings

• The proposed method achieves state-of-the-art performance on the Humaneva-I and Human3.6M benchmarks, outperforming previous methods in 3D pose estimation accuracy.
• On the Humaneva-I dataset, the method achieves a mean 3D error of 104.8 mm under Protocol #3, significantly improving over prior approaches.
• The method generalizes well to unconstrained images: on the LSP dataset, the 2D reprojection error of the estimated 3D poses is 6.76 pixels on average, compared to 9.08 when using CPM detections alone.
• Even with 20-pixel standard deviation Gaussian noise added to 2D detections, the 3D error increases gradually and remains within reasonable bounds, indicating strong robustness to detection noise.
• The fully convolutional network variant successfully hypothesizes occluded limbs, with consistent performance on structured and random occlusion scenarios, though performance degrades slightly on rare poses like 'Sit' or 'Photo'.
• Visual inspection of results on LSP images confirms that the method produces plausible 3D poses even for novel actions and viewpoints not seen during training.