[Paper Review] PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

Shunsuke Saito, Zeng Huang+4 more|arXiv (Cornell University)|May 13, 2019

3D Shape Modeling and Analysis75 references147 citations

TL;DR

The paper presents a fully-convolutional, pixel-aligned implicit function (PIFu) that digitizes high-resolution clothed humans, including textures, from a single image and can incorporate multiple views.

ABSTRACT

We introduce Pixel-aligned Implicit Function (PIFu), a highly effective implicit representation that locally aligns pixels of 2D images with the global context of their corresponding 3D object. Using PIFu, we propose an end-to-end deep learning method for digitizing highly detailed clothed humans that can infer both 3D surface and texture from a single image, and optionally, multiple input images. Highly intricate shapes, such as hairstyles, clothing, as well as their variations and deformations can be digitized in a unified way. Compared to existing representations used for 3D deep learning, PIFu can produce high-resolution surfaces including largely unseen regions such as the back of a person. In particular, it is memory efficient unlike the voxel representation, can handle arbitrary topology, and the resulting surface is spatially aligned with the input image. Furthermore, while previous techniques are designed to process either a single image or multiple views, PIFu extends naturally to arbitrary number of views. We demonstrate high-resolution and robust reconstructions on real world images from the DeepFashion dataset, which contains a variety of challenging clothing types. Our method achieves state-of-the-art performance on a public benchmark and outperforms the prior work for clothed human digitization from a single image.

Motivation & Objective

• Motivate high-quality 3D clothed human reconstruction from a single image or multiple views.
• Develop a fully-convolutional implicit representation that preserves spatial details via pixel-aligned features.
• Enable texture prediction and handle garments with arbitrary topology without templates.
• Demonstrate improvements over prior global-feature implicit approaches and template-based methods.

Proposed method

• Use a fully-convolutional network to learn pixel-aligned implicit fields.
• Regress 3D occupancy/mesh information from global features combined with pixel-aligned local features.
• Support texture inference on 3D surfaces with arbitrary topology using the same framework.
• Incorporate multi-view information when available to improve reconstruction quality.
• Compare against global implicit function baselines (e.g., IM-GAN) and template-based methods.
• Provide technical details and training procedures in supplemental materials and release code.

Experimental results

Research questions

• RQ1 Can a pixel-aligned, fully-convolutional implicit representation recover high-resolution clothed human shapes and textures from a single image?
• RQ2 Does incorporating multi-view information further improve 3D reconstruction quality and texture fidelity compared to single-view input?
• RQ3 How does PIFu compare to global-feature implicit methods and template-based approaches in terms of geometry and texture accuracy?
• RQ4 Can the framework reconstruct garments with arbitrary topology (e.g., dresses, skirts) without relying on templates?
• RQ5 What is the impact of occlusions and profile views on the reconstruction quality?

Key findings

• PIFu enables high-resolution clothed human digitization with texture from a single image and can utilize multiple views.
• The pixel-aligned, fully-convolutional approach improves spatial alignment and detail preservation compared to global implicit methods (IM-GAN).
• PIFu outperforms a template-based method using 3 views on all three metrics shown in the table (Normal, P2S, Chamfer).
• For the template-based method versus PIFu with 3 views, Normal improves from 0.127 to 0.107, P2S from 0.820 to 0.665, and Chamfer from 0.795 to 0.641.
• PIFu can predict texture on 3D surfaces with arbitrary topology and recover globally plausible shapes even under occlusion or profile views.