[Paper Review] Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction

Tong He, John Collomosse+2 more|arXiv (Cornell University)|Jun 15, 2020

3D Shape Modeling and AnalysisEngineering36 references100 citations

TL;DR

Geo-PIFu learns geometry-aligned latent voxel features plus pixel-aligned image features to reconstruct clothed human meshes from a single image, achieving state-of-the-art global topology and local surface detail by regularizing with a latent 3D voxel proxy.

ABSTRACT

We propose Geo-PIFu, a method to recover a 3D mesh from a monocular color image of a clothed person. Our method is based on a deep implicit function-based representation to learn latent voxel features using a structure-aware 3D U-Net, to constrain the model in two ways: first, to resolve feature ambiguities in query point encoding, second, to serve as a coarse human shape proxy to regularize the high-resolution mesh and encourage global shape regularity. We show that, by both encoding query points and constraining global shape using latent voxel features, the reconstruction we obtain for clothed human meshes exhibits less shape distortion and improved surface details compared to competing methods. We evaluate Geo-PIFu on a recent human mesh public dataset that is $10 imes$ larger than the private commercial dataset used in PIFu and previous derivative work. On average, we exceed the state of the art by $42.7\%$ reduction in Chamfer and Point-to-Surface Distances, and $19.4\%$ reduction in normal estimation errors.

Motivation & Objective

• Motivate improved single-view clothed human reconstruction by addressing feature ambiguity and global shape regularity.
• Introduce geometry-aligned latent voxel features trained with a coarse occupancy proxy to regularize high-resolution mesh output.
• Fuse 3D geometry features and 2D pixel features to improve both global topology and local surface details.

Proposed method

• Encode query points with fused geometry-aligned 3D features and pixel-aligned 2D features.
• Lift input image to a low-resolution latent 3D voxel grid using 3D U-Nets to produce geometry-aligned features.
• Extract pixel-aligned features from a 2D U-Net and interpolate them at the projected pixel coordinates of query points.
• Use trilinear interpolation over latent voxel features to obtain geometry-aligned encodings for each query point.
• Train with a coarse occupancy volume loss to supervise latent voxel features and a high-frequency query-point loss for occupancy values.
• Optimize an implicit surface function f(I, P) that maps image I and 3D point P to occupancy σ, using L_geo and L_query losses with a staged training schedule.

Experimental results

Research questions

• RQ1 Can geometry-aligned latent voxel features coupled with pixel-aligned features resolve local feature ambiguity in single-view clothed human reconstruction?
• RQ2 Does a global shape proxy via latent voxel features improve the plausibility and topology of the reconstructed mesh?
• RQ3 How do different feature fusion strategies affect global topology versus local surface details in Geo-PIFu?

Key findings

• Geo-PIFu achieves a 42.7% reduction in Chamfer Distance and Point-to-Surface Distance over the prior state-of-the-art on the DeepHuman dataset.
• Geo-PIFu reduces normal estimation errors by 19.4% on average compared with PIFu.
• Using both geometry-aligned 3D features and pixel-aligned 2D features yields better global topology and local surface details than using either alone.
• A coarse latent voxel proxy regularizes global shape, reducing artifacts such as distorted hands and feet.
• Ablation studies show that early fusion of 3D and 2D features provides competitive performance with simple implementation.