[Paper Review] PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization

Shunsuke Saito, Tomas Simon+2 more|arXiv (Cornell University)|Apr 1, 2020

Human Pose and Action Recognition46 references42 citations

TL;DR

PIFuHD introduces a two-level, end-to-end pixel-aligned implicit function to reconstruct high-fidelity 3D clothed humans from a single 1024×1024 image at 1k resolution, aided by predicted front/back normals to resolve occlusion.

ABSTRACT

Recent advances in image-based 3D human shape estimation have been driven by the significant improvement in representation power afforded by deep neural networks. Although current approaches have demonstrated the potential in real world settings, they still fail to produce reconstructions with the level of detail often present in the input images. We argue that this limitation stems primarily form two conflicting requirements; accurate predictions require large context, but precise predictions require high resolution. Due to memory limitations in current hardware, previous approaches tend to take low resolution images as input to cover large spatial context, and produce less precise (or low resolution) 3D estimates as a result. We address this limitation by formulating a multi-level architecture that is end-to-end trainable. A coarse level observes the whole image at lower resolution and focuses on holistic reasoning. This provides context to an fine level which estimates highly detailed geometry by observing higher-resolution images. We demonstrate that our approach significantly outperforms existing state-of-the-art techniques on single image human shape reconstruction by fully leveraging 1k-resolution input images.

Motivation & Objective

• Motivate high-fidelity single-image 3D human reconstruction beyond 512×512 input limits.
• Propose a multi-level, end-to-end framework that fuses coarse global context with high-resolution local detail.
• Leverage backside/occluded-region guidance via image-space normals to improve completeness and realism.
• Demonstrate 1k-resolution reconstructions that preserve fingers, facial features, and clothing folds.

Proposed method

• Extend Pixel-Aligned Implicit Function (PIFu) to a two-level architecture that processes 512×512 (coarse) and 1024×1024 (fine) inputs.
• Coarse level predicts occupancy using features from downsampled image plus predicted front/back normals; provides 3D embedding to the fine level.
• Fine level uses high-resolution features and the coarse 3D embedding to predict a higher-fidelity occupancy field.
• Front/back normals are generated with an image-to-image translation network (pix2pixHD) and fed as additional features to guide geometry in occluded regions.
• Training uses a multi-level loss with an extended BCE over sampled points, including curvature-aware importance sampling to sharpen details.
• Alternating training of coarse and fine modules can yield higher accuracy than joint end-to-end training.

Experimental results

Research questions

• RQ1 Can a multi-level, pixel-aligned implicit representation leverage high-resolution inputs to produce 1k-detail 3D humans from a single image?
• RQ2 Does incorporating 3D embeddings from a coarse level and image-space normal priors improve back-side and fine-detail reconstructions?
• RQ3 Is alternating training of coarse and fine levels more effective than end-to-end joint training for high-fidelity reconstructions?
• RQ4 How does the approach compare to state-of-the-art single-view methods in terms of point-to-surface distance, Chamfer distance, and normal consistency?

Key findings

• Achieves higher fidelity 3D reconstructions at 1k resolution than single-level PIFu and other baselines on RenderPeople and BUFF datasets.
• Large high-resolution feature embeddings (512×512) plus holistic 3D embeddings improve detail over 128×128 features.
• Incorporating predicted backside normals via image-space guidance reduces ambiguity and improves occluded-region detail.
• Alternating coarse/fine training generally yields higher accuracy than joint end-to-end training.
• Qualitative results show sharper details and more realistic wrinkles in both front and back surfaces when normals are used.