[Paper Review] MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild

Grégory Rogez, Cordelia Schmid|arXiv (Cornell University)|Jul 7, 2016

Human Pose and Action Recognition43 references194 citations

TL;DR

The paper introduces an image-based synthesis engine that uses MoCap data to augment real 2D-pose images with 3D pose annotations, and trains a K-way CNN classifier for full-body 3D pose estimation that outperforms prior methods in controlled settings and shows promise on in-the-wild images.

ABSTRACT

This paper addresses the problem of 3D human pose estimation in the wild. A significant challenge is the lack of training data, i.e., 2D images of humans annotated with 3D poses. Such data is necessary to train state-of-the-art CNN architectures. Here, we propose a solution to generate a large set of photorealistic synthetic images of humans with 3D pose annotations. We introduce an image-based synthesis engine that artificially augments a dataset of real images with 2D human pose annotations using 3D Motion Capture (MoCap) data. Given a candidate 3D pose our algorithm selects for each joint an image whose 2D pose locally matches the projected 3D pose. The selected images are then combined to generate a new synthetic image by stitching local image patches in a kinematically constrained manner. The resulting images are used to train an end-to-end CNN for full-body 3D pose estimation. We cluster the training data into a large number of pose classes and tackle pose estimation as a K-way classification problem. Such an approach is viable only with large training sets such as ours. Our method outperforms the state of the art in terms of 3D pose estimation in controlled environments (Human3.6M) and shows promising results for in-the-wild images (LSP). This demonstrates that CNNs trained on artificial images generalize well to real images.

Motivation & Objective

• Address the lack of large-scale training data for 3D human pose estimation in the wild.
• Develop an image-based synthesis engine that fuses MoCap 3D poses with real 2D-pose images to create synthetic training data with 3D annotations.
• Train an end-to-end CNN to perform 3D pose estimation as a K-way pose classification problem.
• Demonstrate that CNNs trained on synthetic+real data generalize to real in-the-wild images and outperform prior methods in controlled datasets.

Proposed method

• Use a MoCap-guided mosaic construction to assemble synthetic 2D pose images by selecting and stitching joint-centric image patches from real images.
• Define a pose-aware distance D_j between 2D poses to find per-joint matches that best align with a candidate 3D pose.
• Render a 220x220 synthetic image by constructing probability maps for pixel-wise joint-source matching and applying a kinematically constrained mosaic.
• Apply a novel pose-aware blending step to smooth seams while preserving body regions to avoid artifacts.
• Train an end-to-end CNN classifier (based on AlexNet-like architecture) that clusters 3D poses into K=5000 pose classes and outputs a probability distribution over these classes; evaluate absolute and orientation-aware poses after class prediction.

Experimental results

Research questions

• RQ1 Can MoCap-driven image synthesis generate photorealistic in-the-wild training images with accurate 3D pose annotations?
• RQ2 Does training a CNN on synthetic+real data improve 3D pose estimation compared to using real data alone or synthetic data alone?
• RQ3 What is the impact of the number of pose classes (K) and the amount of synthetic data on in-the-wild 3D pose performance?
• RQ4 How does the proposed approach compare to state-of-the-art methods on controlled (Human3.6M) and in-the-wild (LSP) datasets?

Key findings

• Training with synthetic data alone yields substantial gains over real data alone, and combining synthetic with real data yields the best results.
• On Human3.6M (P1 protocol), the regressor trained on synthetic data achieves 101.9 mm Abs Error and 97.2 mm Error, while the classifier trained on synthetic data achieves 97.2 mm Abs Error and 88.1 mm Error in the same setting when using 190,000 3D poses; the combination of synthetic+real yields 125.5 mm Abs Error and 88.1 mm Error for the classifier (absolute alignment considered).
• Under P2 protocol, the classifier achieves 87.3 mm 3D error (absolute) versus 121.2 mm for the regressor, when using synthetic+real data.
• On LSP, using MPII+LSP with CMU data and 2 million synthetic images reduces 2D pose error to 11.2 pixels and 3D pose error to 138.0 mm (from higher baselines), while the 2D error on H3.6M improves to 216.5 Abs? No: Abs 216.5 for 3D source combination; 138.0 mm 3D error in P2-like setting; the method remains competitive with 2D pose estimation baselines.
• Fine-tuning a VGG-16 variant further reduces 2D pose error by 2.3 pixels compared to AlexNet-based setup.
• Qualitative results show correct 3D pose estimations and some failure cases due to unseen poses or left-right/front-back confusions.