[Paper Review] Novel View Synthesis for High-fidelity Headshot Scenes

Satoshi Tsutsui, Weijia Mao+4 more|arXiv (Cornell University)|May 31, 2022

3D Shape Modeling and Analysis4 citations

TL;DR

This paper proposes a novel method for high-fidelity novel view synthesis of headshot scenes by fusing Neural Radiance Fields (NeRF) and 3D Morphable Models (3DMM) using a Generative Adversarial Network (GAN)-based Fusion Net. The approach preserves geometric consistency from NeRF and skin details from 3DMM, achieving state-of-the-art results in perceptual quality and fidelity on real-world scenes with diverse lighting, expressions, and backgrounds.

ABSTRACT

Rendering scenes with a high-quality human face from arbitrary viewpoints is a practical and useful technique for many real-world applications. Recently, Neural Radiance Fields (NeRF), a rendering technique that uses neural networks to approximate classical ray tracing, have been considered as one of the promising approaches for synthesizing novel views from a sparse set of images. We find that NeRF can render new views while maintaining geometric consistency, but it does not properly maintain skin details, such as moles and pores. These details are important particularly for faces because when we look at an image of a face, we are much more sensitive to details than when we look at other objects. On the other hand, 3D Morpable Models (3DMMs) based on traditional meshes and textures can perform well in terms of skin detail despite that it has less precise geometry and cannot cover the head and the entire scene with background. Based on these observations, we propose a method to use both NeRF and 3DMM to synthesize a high-fidelity novel view of a scene with a face. Our method learns a Generative Adversarial Network (GAN) to mix a NeRF-synthesized image and a 3DMM-rendered image and produces a photorealistic scene with a face preserving the skin details. Experiments with various real-world scenes demonstrate the effectiveness of our approach. The code will be available on https://github.com/showlab/headshot .

Motivation & Objective

• To address the challenge of preserving high-fidelity skin details—such as pores and moles—during novel view synthesis of human faces, which are critical for perceptual realism.
• To overcome the limitations of NeRF, which maintains geometric consistency but lacks fine facial texture details, and 3DMM, which preserves skin details but suffers from geometric inaccuracies and lacks scene context.
• To develop a method that synergistically combines the strengths of NeRF (accurate geometry and scene context) and 3DMM (detailed facial textures) for photorealistic novel view synthesis.
• To achieve artifact-free fusion of NeRF-rendered scenes and 3DMM-rendered faces, avoiding visible blending artifacts from naive pasting.

Proposed method

• The method uses a GAN-based Fusion Net that takes as input a novel view synthesized by NeRF and a 3DMM-rendered frontal face image.
• The Fusion Net is trained adversarially to generate a photorealistic output that combines the geometric consistency of NeRF with the skin detail fidelity of 3DMM.
• The 3DMM provides a prior-based, high-resolution texture map of the face, while NeRF provides a neural implicit representation of the full scene with background.
• The fusion process is end-to-end differentiable, enabling joint optimization of the NeRF and 3DMM components through adversarial loss and perceptual loss.
• The training involves optimizing the Fusion Net to minimize perceptual and adversarial losses, ensuring realistic facial textures and consistent blending with the scene.
• The method is evaluated on real-world scenes with varying expressions, lighting, and backgrounds, including indoor, outdoor, and speaking face sequences.

Experimental results

Research questions

• RQ1 Can a hybrid approach combining NeRF and 3DMM achieve superior novel view synthesis for headshot scenes compared to using either method alone?
• RQ2 How can the geometric consistency of NeRF and the skin detail fidelity of 3DMM be effectively fused without introducing visible artifacts or blending errors?
• RQ3 To what extent does adversarial training in a GAN-based fusion network improve the perceptual quality and realism of synthesized headshot images?
• RQ4 Does the proposed method maintain performance across diverse real-world conditions, including complex backgrounds, dynamic expressions, and varying lighting?
• RQ5 What is the relative contribution of the 3DMM prior and the Fusion Net to the final image quality, and how do they compare to naive fusion or ablated models?

Key findings

• The proposed method achieved a mean user study rating of 3.331 ± 0.134, significantly outperforming D-NeRF (3.075 ± 0.094), NerFace (2.569 ± 0.125), and PoseGAN (1.006 ± 0.012), confirming superior perceptual quality.
• In quantitative metrics, the method achieved a mean PSNR of 28.84, SSIM of 0.8263, and LPIPS of 0.0826 across all scenes, outperforming baselines in both fidelity and perceptual similarity.
• Ablation studies showed that removing the 3DMM component degraded performance (LPIPS: 0.0938 vs. 0.0826), confirming its critical role in preserving skin details.
• Removing the Fusion Net led to a significant drop in performance (LPIPS: 0.1425 vs. 0.0826), demonstrating that the GAN-based fusion is essential for artifact-free blending.
• The method consistently outperformed all baselines in LPIPS across all scenes, with the lowest average LPIPS of 0.0826, indicating strong alignment with human perception.
• The ablation study confirmed that both the 3DMM and the Fusion Net are necessary, as removing either degraded the results, with the Fusion Net being crucial for seamless integration.