[Paper Review] 3D-Aware Encoding for Style-based Neural Radiance Fields

Yu-Jhe Li, Tao Xu+7 more|arXiv (Cornell University)|Nov 12, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes NeRF-3DE, a two-stage encoder framework for 3D-aware style-based neural radiance field inversion, enabling high-fidelity single-image to multiview synthesis. By combining a view-invariant base encoder with identity-preserving refinement, it achieves superior 3D consistency and identity preservation over existing GAN inversion and optimization-based methods, outperforming baselines in both reconstruction and novel-view rendering quality.

ABSTRACT

We tackle the task of NeRF inversion for style-based neural radiance fields, (e.g., StyleNeRF). In the task, we aim to learn an inversion function to project an input image to the latent space of a NeRF generator and then synthesize novel views of the original image based on the latent code. Compared with GAN inversion for 2D generative models, NeRF inversion not only needs to 1) preserve the identity of the input image, but also 2) ensure 3D consistency in generated novel views. This requires the latent code obtained from the single-view image to be invariant across multiple views. To address this new challenge, we propose a two-stage encoder for style-based NeRF inversion. In the first stage, we introduce a base encoder that converts the input image to a latent code. To ensure the latent code is view-invariant and is able to synthesize 3D consistent novel view images, we utilize identity contrastive learning to train the base encoder. Second, to better preserve the identity of the input image, we introduce a refining encoder to refine the latent code and add finer details to the output image. Importantly note that the novelty of this model lies in the design of its first-stage encoder which produces the closest latent code lying on the latent manifold and thus the refinement in the second stage would be close to the NeRF manifold. Through extensive experiments, we demonstrate that our proposed two-stage encoder qualitatively and quantitatively exhibits superiority over the existing encoders for inversion in both image reconstruction and novel-view rendering.

Motivation & Objective

• To address the challenge of inverting a single image into a 3D-aware latent space for style-based neural radiance fields (NeRFs), which requires both identity preservation and 3D view consistency.
• To overcome limitations of existing GAN inversion methods when applied to NeRFs, where latent codes often fail to maintain view-invariance and lie outside the valid latent manifold.
• To design a two-stage encoder that first learns a view-invariant latent code via identity contrastive learning, then refines it with fine details to improve image fidelity.
• To ensure the latent code lies close to the NeRF manifold, enabling effective refinement and stable optimization.
• To demonstrate the utility of the learned latent code as an effective initialization for online optimization methods such as PTI and W+ optimization.

Proposed method

• A two-stage encoder architecture: a base encoder maps the input image to a latent code in the W space, followed by a refining encoder that enhances identity details.
• The base encoder is trained using identity contrastive learning to enforce view-invariance across multiple views of the same object.
• Feature-level losses (L_feat) are applied between real and synthesized images to preserve structural and semantic consistency.
• A tri-plane feature loss (L_tri) is introduced to enhance 3D structure and identity preservation during training.
• The model is trained using both real images and synthesized views, with the latter used to supervise latent code generalization across views.
• The framework supports integration with online optimization techniques by providing a high-quality initial latent code for PTI and W+ optimization.

Experimental results

Research questions

• RQ1 Can a two-stage encoder architecture effectively learn a 3D-consistent and identity-preserving latent code from a single image for style-based NeRFs?
• RQ2 How does identity contrastive learning in the base encoder improve view-invariance and 3D consistency compared to standard GAN inversion methods?
• RQ3 To what extent does refining the latent code with a second-stage encoder improve identity and detail preservation without compromising 3D structure?
• RQ4 Can the learned latent code serve as a better initialization than random or baseline encoders for online optimization methods like PTI or W+ optimization?
• RQ5 What is the relative contribution of real images, synthesized views, and feature-level losses in achieving robust NeRF inversion?

Key findings

• The proposed NeRF-3DE achieves the highest identity (ID) score among all baselines across four yaw angles, outperforming even PTI, which excels in reconstruction but fails in identity preservation.
• The model with both real and synthesized images and full feature losses (L_feat) achieves the best balance of 3D consistency and identity fidelity, with ablation showing that excluding any component degrades performance.
• Using only synthesized images (without real images) leads to a loss of identity preservation, demonstrating the necessity of real image supervision for identity alignment.
• The tri-plane feature loss (L_tri) plays a more critical role than other components in preserving 3D structure and identity, as shown in ablation studies.
• The two-stage encoder design significantly improves novel-view rendering quality, with the refined latent code producing sharper and more consistent views than the base encoder alone.
• The latent code from NeRF-3DE serves as a superior initialization for online optimization, accelerating convergence and improving both image quality and 3D consistency in PTI and W+ optimization.