[Paper Review] View-Invariant Probabilistic Embedding for Human Pose

Jennifer J. Sun, Jiaping Zhao+4 more|arXiv (Cornell University)|Dec 2, 2019

Human Pose and Action Recognition63 references4 citations

TL;DR

This paper proposes Pr-VIPE, a probabilistic view-invariant pose embedding model that learns 2D keypoint-based representations invariant to viewpoint changes without predicting 3D poses. By modeling embeddings as multivariate Gaussians, it captures input uncertainty from 2D-to-3D ambiguity, achieving state-of-the-art performance in cross-view pose retrieval and enabling applications in action recognition and video alignment using only 2D keypoints.

ABSTRACT

Depictions of similar human body configurations can vary with changing viewpoints. Using only 2D information, we would like to enable vision algorithms to recognize similarity in human body poses across multiple views. This ability is useful for analyzing body movements and human behaviors in images and videos. In this paper, we propose an approach for learning a compact view-invariant embedding space from 2D joint keypoints alone, without explicitly predicting 3D poses. Since 2D poses are projected from 3D space, they have an inherent ambiguity, which is difficult to represent through a deterministic mapping. Hence, we use probabilistic embeddings to model this input uncertainty. Experimental results show that our embedding model achieves higher accuracy when retrieving similar poses across different camera views, in comparison with 2D-to-3D pose lifting models. We also demonstrate the effectiveness of applying our embeddings to view-invariant action recognition and video alignment. Our code is available at https://github.com/google-research/google-research/tree/master/poem.

Motivation & Objective

• To learn a compact, view-invariant embedding space for 2D human poses that remains consistent across different camera viewpoints.
• To address the inherent ambiguity in 2D pose projections by modeling input uncertainty through probabilistic embeddings.
• To enable accurate pose retrieval and downstream vision tasks (e.g., action recognition, video alignment) using only 2D keypoint inputs without 3D ground truth or image context.
• To demonstrate that 2D keypoint embeddings alone can achieve view invariance without explicit 3D pose prediction or rigid alignment post-processing.

Proposed method

• The model uses a deep neural network to map 2D keypoint coordinates into a probabilistic embedding space defined by multivariate Gaussian distributions (mean and covariance).
• The embedding loss is optimized using a contrastive loss with hard negative mining, encouraging similar poses from different views to be close in embedding space.
• Training data is augmented by synthesizing multi-view 2D poses from 3D poses via perspective projection, improving generalization to diverse viewpoints.
• The model is trained end-to-end on 2D keypoint detections from datasets like Human3.6M and 3DHP, without requiring image-level supervision or 3D ground truth during inference.
• The probabilistic formulation allows the model to learn uncertainty: higher variance in the embedding correlates with higher 2D pose ambiguity.
• For downstream tasks, similarity is computed via Mahalanobis distance in the embedding space, enabling efficient retrieval and alignment.

Experimental results

Research questions

• RQ1 Can 2D pose embeddings alone achieve view-invariant similarity without predicting 3D poses?
• RQ2 How can input uncertainty from 2D-to-3D projection ambiguity be effectively modeled in the embedding space?
• RQ3 Does a probabilistic embedding formulation improve performance on cross-view pose retrieval compared to deterministic embeddings?
• RQ4 Can the learned embeddings generalize to in-the-wild data and enable effective action recognition and video alignment?

Key findings

• Pr-VIPE achieves a 75.2% Hit@1 on the H3.6M evaluation subset using only 2D keypoint detections and no ground truth, outperforming fully supervised EpipolarPose (72.7% Hit@1).
• The model generalizes well to unseen poses and views, as shown by accurate retrieval on 3DHP and in-the-wild hold-out sets despite diverse subjects and camera angles.
• The probabilistic embedding formulation correlates with input ambiguity: higher variance in the embedding is observed for 2D poses with greater 3D projection ambiguity.
• Pr-VIPE enables effective video alignment on the Penn Action dataset, synchronizing action videos from different views by matching pose embeddings without explicit temporal alignment.
• The model achieves superior retrieval accuracy compared to 2D-to-3D lifting models, and unlike those models, does not require rigid 3D alignment between query and index pairs.
• Qualitative results show that high-confidence retrievals are visually accurate, even across large viewpoint changes and partial occlusions, validating the model’s robustness.