[Paper Review] Exemplar Fine-Tuning for 3D Human Pose Fitting Towards In-the-Wild 3D Human Pose Estimation

Hanbyul Joo, Natalia Neverova+1 more|arXiv (Cornell University)|Apr 7, 2020

Human Pose and Action Recognition69 references51 citations

TL;DR

This paper introduces Exemplar Fine-Tuning (EFT), a method to generate accurate 3D human pose annotations from 2D keypoint datasets like COCO and MPII by leveraging a 3D parametric body model and data-driven pose priors to resolve depth ambiguity. The resulting large-scale in-the-wild 3D dataset enables state-of-the-art 3D human pose estimation, even on challenging outdoor and Internet videos.

ABSTRACT

We propose a method for building large collections of human poses with full 3D annotations captured `in the wild', for which specialized capture equipment cannot be used. We start with a dataset with 2D keypoint annotations such as COCO and MPII and generates corresponding 3D poses. This is done via Exemplar Fine-Tuning (EFT), a new method to fit a 3D parametric model to 2D keypoints. EFT is accurate and can exploit a data-driven pose prior to resolve the depth reconstruction ambiguity that comes from using only 2D observations as input. We use EFT to augment these large in-the-wild datasets with plausible and accurate 3D pose annotations. We then use this data to strongly supervise a 3D pose regression network, achieving state-of-the-art results in standard benchmarks, including the ones collected outdoor. This network also achieves unprecedented 3D pose estimation quality on extremely challenging Internet videos.

Motivation & Objective

• To address the lack of large-scale, fully 3D-annotated in-the-wild human pose datasets due to impracticality of specialized capture equipment.
• To resolve the depth ambiguity inherent in 2D-only keypoint observations when reconstructing 3D poses.
• To develop a method that generates plausible and accurate 3D pose annotations for unconstrained, real-world videos.
• To improve 3D human pose estimation performance on challenging, uncontrolled environments such as outdoor scenes and Internet videos.
• To leverage data-driven priors to enhance the realism and accuracy of 3D pose reconstructions from 2D supervision alone.

Proposed method

• Exemplar Fine-Tuning (EFT) is proposed as a novel optimization-based method to fit a 3D parametric body model (e.g., SMPL) to 2D keypoint detections.
• EFT incorporates a data-driven pose prior learned from existing 3D human pose data to guide the 3D reconstruction and resolve depth ambiguity.
• The method optimizes the 3D joint positions and body shape parameters by minimizing a differentiable loss that combines 2D keypoint reprojection and pose prior regularization.
• EFT is applied at scale to existing 2D keypoint datasets (e.g., COCO, MPII) to generate large collections of 3D-annotated in-the-wild images.
• The resulting synthetic 3D-annotated dataset is used to supervise a 3D pose regression network, improving its generalization to unconstrained settings.
• The final model is trained and evaluated on standard benchmarks, including in-the-wild and outdoor datasets, achieving state-of-the-art performance.

Experimental results

Research questions

• RQ1 Can a data-driven pose prior effectively resolve depth ambiguity in 2D-to-3D pose lifting without specialized 3D capture?
• RQ2 Can Exemplar Fine-Tuning generate high-quality, realistic 3D human poses from 2D keypoint annotations in unconstrained, real-world settings?
• RQ3 To what extent does fine-tuning with EFT-generated 3D data improve 3D pose estimation performance on challenging in-the-wild and outdoor benchmarks?
• RQ4 Can a 3D regression network trained on EFT-annotated data generalize to extremely challenging Internet videos with complex poses and occlusions?
• RQ5 How does the quality of EFT-generated 3D annotations compare to real 3D annotations in terms of downstream 3D pose estimation accuracy?

Key findings

• Exemplar Fine-Tuning (EFT) successfully generates accurate and plausible 3D human poses from 2D keypoint detections in unconstrained, in-the-wild settings.
• The EFT-generated 3D dataset enables strong supervision of a 3D pose regression network, resulting in state-of-the-art performance on standard benchmarks including in-the-wild and outdoor datasets.
• The method achieves unprecedented 3D pose estimation quality on extremely challenging Internet videos, demonstrating robustness to complex scenes and occlusions.
• The integration of a data-driven pose prior in EFT significantly improves depth estimation accuracy by resolving the inherent ambiguity in 2D observations.
• The resulting 3D-annotated dataset from EFT is large-scale and suitable for training deep networks to generalize beyond controlled laboratory settings.
• The final 3D pose estimation model outperforms prior methods on standard evaluation protocols, particularly in real-world and unconstrained environments.