[Paper Review] Generating Multiple Hypotheses for 3D Human Pose Estimation with Mixture Density Network

Chen Li, Gim Hee Lee|arXiv (Cornell University)|Apr 11, 2019

Human Pose and Action Recognition29 references21 citations

TL;DR

This paper proposes a mixture density network (MDN) to generate multiple geometrically feasible 3D human pose hypotheses from 2D joint inputs, addressing depth ambiguity and occlusion in monocular 3D pose estimation. By modeling the 3D pose distribution as a mixture of Gaussians, the method produces multiple consistent hypotheses that reproject accurately to the input 2D joints, achieving state-of-the-art performance on Human3.6M and strong generalization on MPII and MPI-INF-3DHP datasets.

ABSTRACT

3D human pose estimation from a monocular image or 2D joints is an ill-posed problem because of depth ambiguity and occluded joints. We argue that 3D human pose estimation from a monocular input is an inverse problem where multiple feasible solutions can exist. In this paper, we propose a novel approach to generate multiple feasible hypotheses of the 3D pose from 2D joints.In contrast to existing deep learning approaches which minimize a mean square error based on an unimodal Gaussian distribution, our method is able to generate multiple feasible hypotheses of 3D pose based on a multimodal mixture density networks. Our experiments show that the 3D poses estimated by our approach from an input of 2D joints are consistent in 2D reprojections, which supports our argument that multiple solutions exist for the 2D-to-3D inverse problem. Furthermore, we show state-of-the-art performance on the Human3.6M dataset in both best hypothesis and multi-view settings, and we demonstrate the generalization capacity of our model by testing on the MPII and MPI-INF-3DHP datasets. Our code is available at the project website.

Motivation & Objective

• To address the inherent ambiguity in 3D human pose estimation from monocular images or 2D joints, where multiple 3D poses can project to the same 2D joint configuration.
• To move beyond single-estimate approaches that minimize mean squared error under unimodal Gaussian assumptions, which may overfit and fail to capture uncertainty.
• To model the multimodal nature of feasible 3D poses using a mixture density network (MDN), enabling the generation of multiple consistent hypotheses.
• To improve generalization to unseen domains and challenging poses, including occluded joints, by learning uncertainty-aware pose distributions.
• To demonstrate state-of-the-art performance in both best-hypothesis and multi-view settings on benchmark datasets.

Proposed method

• The method employs a two-stage framework: a 2D pose estimator provides input joints, followed by a 3D pose hypotheses generator based on a mixture density network (MDN).
• The MDN models the 3D pose distribution as a mixture of M Gaussian kernels, with outputs including mixing coefficients, means, and variances for each kernel.
• The network is trained by minimizing the negative log-likelihood of the mixture-of-Gaussians distribution, which better captures multimodal uncertainty than unimodal regression.
• A Dirichlet conjugate prior is applied to the mixing coefficients to prevent overfitting to a single Gaussian kernel, especially on rare or ambiguous poses.
• The 3D pose hypotheses are generated as the means of the M Gaussian kernels, and their 2D reprojections are evaluated for consistency with the input 2D joints.
• The model is trained end-to-end using linear layers with non-linear activation functions, and no additional 2D consistency constraints are enforced during training.

Experimental results

Research questions

• RQ1 Can a deep learning model generate multiple plausible 3D human pose hypotheses that are consistent with the same 2D joint input, reflecting the inherent ambiguity of the 3D reconstruction problem?
• RQ2 Does modeling the 3D pose distribution as a mixture of Gaussians (MDN) lead to better performance and uncertainty estimation than unimodal regression with mean squared error?
• RQ3 How well does the proposed model generalize to datasets without 3D training labels, such as MPII and MPI-INF-3DHP?
• RQ4 To what extent do the generated hypotheses remain consistent in 2D reprojection, even without explicit 2D consistency loss?
• RQ5 How does the inclusion of a Dirichlet prior improve robustness on rare or ambiguous poses like 'SittingDown' or 'GettingDown'?

Key findings

• The proposed MDN-based method achieves state-of-the-art performance on the Human3.6M dataset in both best-hypothesis and multi-view settings, outperforming prior methods.
• The 3D pose hypotheses generated by the model show high consistency in 2D reprojection, with a PCKh@0.5 score close to 1.0, indicating that all hypotheses project nearly identically to the input 2D joints.
• On the MPII and MPI-INF-3DHP datasets, where no 3D training data was used, the model still generalizes well, with 3DPCK results slightly below but comparable to those of models trained on the same data.
• The ablation study shows that using M=5 Gaussian kernels provides the best trade-off between performance and computational cost, as performance plateaus beyond M=5.
• The Dirichlet prior significantly improves performance on challenging poses such as 'Sitting' and 'SittingDown', reducing overfitting to dominant 'standing' poses in the training data.
• The model generates distinct hypotheses only for ambiguous poses (e.g., 'GettingDown', 'SittingDown'), while simpler poses like 'Standing' yield similar hypotheses, confirming the model captures uncertainty appropriately.