[Paper Review] FENeRF: Face Editing in Neural Radiance Fields

Jingxiang Sun, Xuan Wang+5 more|arXiv (Cornell University)|Nov 30, 2021

Face recognition and analysis4 citations

TL;DR

FENeRF proposes a 3D-aware, locally editable portrait generation framework using disentangled latent codes for shape and texture, jointly optimizing semantic and texture fields in a spatially-aligned 3D volume via GAN inversion. It achieves view-consistent, high-fidelity editing using only monocular images and paired semantic masks, outperforming SOTA in editing fidelity and geometry accuracy.

ABSTRACT

Previous portrait image generation methods roughly fall into two categories: 2D GANs and 3D-aware GANs. 2D GANs can generate high fidelity portraits but with low view consistency. 3D-aware GAN methods can maintain view consistency but their generated images are not locally editable. To overcome these limitations, we propose FENeRF, a 3D-aware generator that can produce view-consistent and locally-editable portrait images. Our method uses two decoupled latent codes to generate corresponding facial semantics and texture in a spatial aligned 3D volume with shared geometry. Benefiting from such underlying 3D representation, FENeRF can jointly render the boundary-aligned image and semantic mask and use the semantic mask to edit the 3D volume via GAN inversion. We further show such 3D representation can be learned from widely available monocular image and semantic mask pairs. Moreover, we reveal that joint learning semantics and texture helps to generate finer geometry. Our experiments demonstrate that FENeRF outperforms state-of-the-art methods in various face editing tasks.

Motivation & Objective

• To address the limitations of 2D GANs in view inconsistency and 3D-aware GANs in lack of local editability for portrait synthesis.
• To enable interactive, local editing of free-view portraits while maintaining strict view consistency.
• To learn a 3D-aware, spatially-aligned representation of facial semantics, geometry, and texture from monocular images and semantic masks.
• To improve 3D geometry quality through joint learning of semantics and texture.
• To enable downstream applications such as style transfer, attribute editing, and style mixing in 3D free-view settings.

Proposed method

• The generator uses two decoupled latent codes, $\mathbf{Z}_s$ for shape and $\mathbf{Z}_t$ for texture, to produce a shared 3D volume with spatially-aligned density, semantic, and texture fields.
• A learnable coordinate embedding $e_{\text{coord}}$ is injected into the MLP to preserve high-frequency details in the rendered images.
• The model employs a dual discriminator setup: a color discriminator $D_c$ for image realism and a semantic discriminator $D_s$ for content alignment between images and semantic maps.
• GAN inversion is used to invert real images into latent codes, enabling local editing via manipulation of the semantic mask in the 3D volume.
• Joint training with both image and semantic supervision ensures spatial alignment and improves geometry and detail quality.
• The method trains on monocular images with paired semantic masks, avoiding the need for multi-view or 3D supervision.

Experimental results

Research questions

• RQ1 Can a 3D-aware GAN generator achieve both view consistency and local editability in portrait synthesis?
• RQ2 Can monocular images with semantic masks be effectively used to train a 3D-aware portrait generator without multi-view or 3D supervision?
• RQ3 Does joint learning of semantics and texture improve 3D geometry reconstruction quality?
• RQ4 Can semantic masks serve as an effective interface for interactive, localized editing of 3D portrait geometry and texture?
• RQ5 How does the integration of learnable coordinate embeddings affect high-frequency detail preservation in generated portraits?

Key findings

• FENeRF outperforms state-of-the-art methods in view consistency, editing fidelity, and geometry accuracy across multiple benchmarks, including CelebAMask-HQ and FFHQ.
• Joint training with both image and semantic supervision significantly improves geometry quality, as shown by more accurate and smooth reconstructed meshes and depth maps.
• The inclusion of a learnable coordinate embedding $e_{\text{coord}}$ in the color branch enables sharp image details while minimizing artifacts, outperforming early injection or omission.
• Semantic map editing via GAN inversion enables precise, localized manipulation of facial attributes (e.g., nose size, shape) with consistent geometry and texture across views.
• The method successfully supports complex editing tasks such as style mixing, style transfer, and free-view avatar generation with explicit camera control.
• Abllation studies confirm that semantic supervision is essential for reliable geometry learning, and image rendering alone is insufficient to produce accurate facial structures.