[Paper Review] Portrait Neural Radiance Fields from a Single Image

Chen Gao, YiChang Shih+3 more|arXiv (Cornell University)|Dec 10, 2020

Generative Adversarial Networks and Image Synthesis74 references42 citations

TL;DR

The paper presents a method to synthesize portrait Neural Radiance Fields from a single headshot by pretraining a NeRF with meta-learning on a light-stage portrait dataset and finetuning on the input subject within a canonical face space.

ABSTRACT

We present a method for estimating Neural Radiance Fields (NeRF) from a single headshot portrait. While NeRF has demonstrated high-quality view synthesis, it requires multiple images of static scenes and thus impractical for casual captures and moving subjects. In this work, we propose to pretrain the weights of a multilayer perceptron (MLP), which implicitly models the volumetric density and colors, with a meta-learning framework using a light stage portrait dataset. To improve the generalization to unseen faces, we train the MLP in the canonical coordinate space approximated by 3D face morphable models. We quantitatively evaluate the method using controlled captures and demonstrate the generalization to real portrait images, showing favorable results against state-of-the-arts.

Motivation & Objective

• Enable photorealistic view synthesis for portraits from a single image.
• Generalize NeRF to unseen subjects without multiple captures.
• Incorporate face geometry priors via a canonical face space to handle hair, upper head, and torso.
• Provide a multi-view portrait dataset for quantitative evaluation.
• Demonstrate applications such as novel view synthesis and perspective/fov manipulation.

Proposed method

• Pretrain an MLP-based NeRF on a light-stage portrait dataset to learn general face priors.
• Warp world coordinates to a canonical face space using a rigid transform derived from 3D morphable model geometry.
• Fine-tune the pretrained NeRF on the single input frontal view to adapt to the subject (theta_s).
• Render novel views via volume rendering after warping rays to the canonical space.
• Train with gradient-based meta-learning to enable rapid adaptation to unseen subjects.
• Use canonical coordinates to improve generalization and realism across hairstyles, accessories, and upper head regions.

Experimental results

Research questions

• RQ1 Can a NeRF be effectively trained from a single portrait image to synthesize novel views of the same subject?
• RQ2 Does meta-learning plus canonical face space improve generalization to unseen subjects compared to vanilla pretraining?
• RQ3 How does the proposed method compare to state-of-the-art portrait view synthesis methods on controlled and real-world data?
• RQ4 What is the impact of training data size and input-view count on synthesis quality?
• RQ5 What are the limitations and failure modes when applying single-image portrait NeRF to real-world images?

Key findings

• Our method achieves higher PSNR, SSIM, and lower LPIPS than prior portrait view synthesis methods on a light-stage dataset (PSNR 23.92, SSIM 0.7688, LPIPS 0.161).
• Pretraining with meta-learning plus canonical face coordinates substantially improves generalization to unseen subjects versus random initialization or naive pretraining.
• Canonical face space warping reduces artifacts in eyes and chin and yields better quantitative metrics than using world coordinates.
• Finetuning on a single frontal view enables realistic, identity-preserving novel-view synthesis across diverse subjects including hair and accessories.
• The approach scales with more input views at test time, with performance improving from 1 to 5 views (e.g., PSNR from 24.98 to 32.45 in ablations).
• The method provides perspective manipulation capabilities (dollying with focal-length adjustment) due to its 3D neural representation.