[Paper Review] Weakly-Supervised Discovery of Geometry-Aware Representation for 3D Human Pose Estimation

Xipeng Chen, Kwan-Yee Lin+4 more|arXiv (Cornell University)|Mar 21, 2019

Human Pose and Action RecognitionComputer Science43 references42 citations

TL;DR

The paper introduces a weakly supervised framework that learns a geometry-aware 3D pose representation from multi-view 2D skeletons via a skeleton-based view synthesis encoder-decoder, with a representation consistency constraint to improve monocular 3D pose estimation.

ABSTRACT

Recent studies have shown remarkable advances in 3D human pose estimation from monocular images, with the help of large-scale in-door 3D datasets and sophisticated network architectures. However, the generalizability to different environments remains an elusive goal. In this work, we propose a geometry-aware 3D representation for the human pose to address this limitation by using multiple views in a simple auto-encoder model at the training stage and only 2D keypoint information as supervision. A view synthesis framework is proposed to learn the shared 3D representation between viewpoints with synthesizing the human pose from one viewpoint to the other one. Instead of performing a direct transfer in the raw image-level, we propose a skeleton-based encoder-decoder mechanism to distil only pose-related representation in the latent space. A learning-based representation consistency constraint is further introduced to facilitate the robustness of latent 3D representation. Since the learnt representation encodes 3D geometry information, mapping it to 3D pose will be much easier than conventional frameworks that use an image or 2D coordinates as the input of 3D pose estimator. We demonstrate our approach on the task of 3D human pose estimation. Comprehensive experiments on three popular benchmarks show that our model can significantly improve the performance of state-of-the-art methods with simply injecting the representation as a robust 3D prior.

Motivation & Objective

• Motivate robust 3D pose estimation that generalizes across environments and motions by learning geometry-aware representations with limited 3D annotations.
• Learn a shared 3D pose representation from multi-view skeletons using 2D supervision only.
• Distill pose-related information into a latent space that maps more easily to 3D pose.
• Improve generalization by leveraging view synthesis and a latent space consistency constraint.

Proposed method

• Use 2D skeleton maps derived from multi-view images as the input instead of raw images.
• Train a skeleton-based encoder–decoder to synthesize the target-view skeleton from the source-view skeleton, with the latent code representing geometry G.
• Constrain G to be a semantically meaningful 3D pose representation via a representation consistency loss across view directions.
• Introduce a bidirectional encoder–decoder setup to enforce latent-space consistency under known view rotations.
• Inject the learned geometry representation G as a prior into 3D pose regressors, enabling simple regression from G to 3D joint coordinates.

Experimental results

Research questions

• RQ1 Can a geometry-aware 3D representation for human pose be learned from multi-view data using only 2D annotations?
• RQ2 Does a skeleton-based view synthesis framework, coupled with a latent-space consistency constraint, yield a robust 3D pose representation that improves monocular pose estimation?
• RQ3 Can the learned geometry representation serve as an effective prior to boost state-of-the-art 3D pose estimation methods across datasets and protocols?

Key findings

• A skeleton-based view synthesis framework yields a geometry representation G that improves 3D pose estimation when injected as a prior.
• With limited 3D annotations, regressing 3D pose from G from a simple two-layer regressor can achieve reasonable results, and G can boost stronger baselines across protocols.
• A representation consistency constraint reduces implausible poses and improves robustness of G, evidenced by ablation showing improved results when the constraint is included.
• Data augmentation via virtual cameras and the representation consistency constraint together yield observable performance gains over baselines.
• The learned G generalizes across datasets, and qualitative results on in-the-wild MPII demonstrate the method’s practical effectiveness.