[Paper Review] Towards Accurate Markerless Human Shape and Pose Estimation over Time

Yinghao Huang, Federica Bogo+5 more|arXiv (Cornell University)|Jul 24, 2017

Human Pose and Action RecognitionComputer Science7 references68 citations

TL;DR

MuVS extends SMPLify to multi-view and monocular videos by fitting a 3D SMPL body model to 2D joints and silhouettes, with a DCT-based temporal prior, yielding accurate pose and realistic body meshes without background requirements.

ABSTRACT

Existing marker-less motion capture methods often assume known backgrounds, static cameras, and sequence specific motion priors, which narrows its application scenarios. Here we propose a fully automatic method that given multi-view video, estimates 3D human motion and body shape. We take recent SMPLify \cite{bogo2016keep} as the base method, and extend it in several ways. First we fit the body to 2D features detected in multi-view images. Second, we use a CNN method to segment the person in each image and fit the 3D body model to the contours to further improves accuracy. Third we utilize a generic and robust DCT temporal prior to handle the left and right side swapping issue sometimes introduced by the 2D pose estimator. Validation on standard benchmarks shows our results are comparable to the state of the art and also provide a realistic 3D shape avatar. We also demonstrate accurate results on HumanEva and on challenging dance sequences from YouTube in monocular case.

Motivation & Objective

• Provide fully automatic markerless 3D human pose and shape estimation from multi-view videos.
• Leverage SMPL for coherent body shape and pose representation.
• Incorporate CNN-based 2D joints and silhouettes to strengthen fitting accuracy.
• Resolve left-right ambiguities and temporal inconsistencies with a DCT-based temporal prior.
• Demonstrate applicability to monocular sequences and challenging real-world videos.

Proposed method

• Represent the human body with the SMPL model and optimize pose and shape to align with multi-view 2D joints.
• Use CNNs to detect 2D joints and to segment body silhouettes in each view.
• Fit SMPL independently per frame across views, then refine shape with silhouettes (E_S term).
• Combine frames over time using a low-dimensional DCT temporal prior to constrain joint trajectories (E_T terms) and minimize a robust joint reprojection error.
• Adopt a hierarchical optimization strategy: first fit to joints, then incorporate silhouettes and temporal regularization; use robust Geman-McClure error for residuals.
• Provide an ablation study to quantify the impact of silhouettes, temporal prior, and multi-view data; demonstrate monocular capabilities with limited temporal cues.

Experimental results

Research questions

• RQ1 Can a fully automatic multi-view system estimate accurate 3D human pose and body shape without background subtraction or user intervention?
• RQ2 Does integrating silhouette information and a temporal DCT prior improve pose and shape accuracy over frame-by-frame fitting?
• RQ3 How does multi-view data compare to monocular sequences in resolving ambiguities like left-right swaps and pose ambiguities?
• RQ4 Is the method capable of producing realistic 3D body meshes suitable for animation, beyond just joint estimation?
• RQ5 Can the approach generalize to challenging monocular videos and to datasets beyond HumanEva, such as Human3.6M and YouTube sequences?

Key findings

• MuVS achieves pose errors comparable to state-of-the-art benchmarks on HumanEva and Human3.6M.
• Silhouette fitting significantly improves 3D pose and shape accuracy and mesh realism.
• A DCT-based temporal prior reduces frame-to-frame errors and mitigates leg-swap and other temporal artifacts.
• Using multiple views yields substantial improvements over single-view SMPLify, especially in orientation and pose accuracy.
• The method yields realistic body meshes via the SMPL model, enabling believable avatar generation and animation.
• Monocular sequences show promising results, with temporal coherence enabling reasonable reconstructions under limited depth cues.