[Paper Review] Geometric Pose Affordance: 3D Human Pose with Scene Constraints

Zhe Wang, Liyan Chen+3 more|arXiv (Cornell University)|May 19, 2019

Human Pose and Action Recognition39 references41 citations

TL;DR

This work introduces Geometric Pose Affordance (GPA), a dataset and method that use multi-layer depth maps of scene geometry to constrain monocular 3D human pose estimation, improving accuracy under occlusion and complex scenes. It presents two integration approaches—geometry-encoded inputs and a differentiable geometric-consistency loss—and demonstrates improvements over scene-agnostic baselines.

ABSTRACT

Full 3D estimation of human pose from a single image remains a challenging task despite many recent advances. In this paper, we explore the hypothesis that strong prior information about scene geometry can be used to improve pose estimation accuracy. To tackle this question empirically, we have assembled a novel $ extbf{Geometric Pose Affordance}$ dataset, consisting of multi-view imagery of people interacting with a variety of rich 3D environments. We utilized a commercial motion capture system to collect gold-standard estimates of pose and construct accurate geometric 3D CAD models of the scene itself. To inject prior knowledge of scene constraints into existing frameworks for pose estimation from images, we introduce a novel, view-based representation of scene geometry, a $ extbf{multi-layer depth map}$, which employs multi-hit ray tracing to concisely encode multiple surface entry and exit points along each camera view ray direction. We propose two different mechanisms for integrating multi-layer depth information pose estimation: input as encoded ray features used in lifting 2D pose to full 3D, and secondly as a differentiable loss that encourages learned models to favor geometrically consistent pose estimates. We show experimentally that these techniques can improve the accuracy of 3D pose estimates, particularly in the presence of occlusion and complex scene geometry.

Motivation & Objective

• Motivate how strong scene geometry priors can improve 3D human pose estimation from a single image.
• Create and publicize a dataset (GPA) with rich ground-truth pose and scene geometry for pose–scene interaction research.
• Propose a compact multi-layer depth map representation of scene geometry for efficient integration into CNN-based pose models.
• Develop two mechanisms to incorporate geometry into pose estimation: encoded scene geometry as input features and a geometric consistency loss.
• Demonstrate improved pose estimation accuracy, especially under occlusion and complex scene geometry, over scene-agnostic baselines.

Proposed method

• Adopt a ResNet-50 backbone to predict 2D joint heatmaps and depth for 3D pose from cropped images.
• Compute and encode a multi-layer depth map per camera view to capture occluded and visible surface intersections along each view ray.
• Introduce a geometric consistency loss that penalizes pose predictions penetrating scene geometry along view rays.
• Provide two encoding schemes for multi-layer depth as input features: a view-centered cropped depth map and a volumetric around-root encoding.
• Train in stages: 2D pose module, then 3D depth regression, then geometry-aware components added to the model.
• Evaluate against baselines including a simple lifting model and PoseNet-style integral regression.

Experimental results

Research questions

• RQ1 Can explicit scene geometry priors improve monocular 3D human pose estimation in cluttered or occluded environments?
• RQ2 How can multi-layer depth representations be leveraged to constrain joint depths without prohibitive computation?
• RQ3 Do geometry-encoded inputs and differentiable geometric losses yield measurable accuracy gains over scene-agnostic pose models?
• RQ4 How does the GPA dataset support studying human-scene affordances and improve pose estimation under occlusion and near-geometry scenarios?

Key findings

• A novel GPA dataset with ground-truth 3D pose and detailed scene geometry is released to support geometry-aware pose estimation research.
• Two geometry integration approaches (encoded multi-layer depth inputs and geometric consistency loss) improve 3D pose estimation compared to scene-agnostic baselines.
• The strongest gains are observed under occlusion and in scenes with complex geometry, validating the usefulness of scene constraints.
• The multi-layer depth representation efficiently encodes scene geometry and can be integrated as additional CNN input channels or used to constrain predicted depths.