[Paper Review] ED-NeRF: Efficient Text-Guided Editing of 3D Scene with Latent Space NeRF

Jang-Ho Park, Gihyun Kwon+1 more|arXiv (Cornell University)|Oct 4, 2023

Generative Adversarial Networks and Image Synthesis8 citations

TL;DR

ED-NeRF edits real-world 3D scenes by training NeRF in the latent space of a diffusion model, using a refinement layer and Delta Denoising Score (DDS) with a mask to enable efficient, targeted text-guided edits.

ABSTRACT

Recently, there has been a significant advancement in text-to-image diffusion models, leading to groundbreaking performance in 2D image generation. These advancements have been extended to 3D models, enabling the generation of novel 3D objects from textual descriptions. This has evolved into NeRF editing methods, which allow the manipulation of existing 3D objects through textual conditioning. However, existing NeRF editing techniques have faced limitations in their performance due to slow training speeds and the use of loss functions that do not adequately consider editing. To address this, here we present a novel 3D NeRF editing approach dubbed ED-NeRF by successfully embedding real-world scenes into the latent space of the latent diffusion model (LDM) through a unique refinement layer. This approach enables us to obtain a NeRF backbone that is not only faster but also more amenable to editing compared to traditional image space NeRF editing. Furthermore, we propose an improved loss function tailored for editing by migrating the delta denoising score (DDS) distillation loss, originally used in 2D image editing to the three-dimensional domain. This novel loss function surpasses the well-known score distillation sampling (SDS) loss in terms of suitability for editing purposes. Our experimental results demonstrate that ED-NeRF achieves faster editing speed while producing improved output quality compared to state-of-the-art 3D editing models.

Motivation & Objective

• Motivation to enable fast, editing-friendly 3D NeRF models using latent diffusion priors.
• Develop a latent-space NeRF training pipeline to embed real-world scenes for faster editing.
• Propose a refinement layer to improve latent-space rendering quality.
• Introduce a mask-guided Delta Denoising Score (DDS) loss to guide edits and suppress unintended changes.
• Demonstrate improved editing speed and output quality over state-of-the-art 3D editing methods.

Proposed method

• Train NeRF directly in the latent space of Stable Diffusion by encoding real-world multi-view images into latent feature maps and predicting latent features rather than RGB colors.
• Add a refinement layer with ResNet/self-attention blocks to capture inter-pixel interactions and improve novel view synthesis.
• Use a total loss combining reconstruction in latent space and refinement loss to align NeRF outputs with latent encodings across views.
• Extend Delta Denoising Score (DDS) to 3D by computing score differences between source and target prompts and applying a masked DDS to local edits.
• Incorporate a masked reconstruction loss to constrain updates outside the target region and preserve overall structure.
• Optional: render edited latents via the latent decoder to obtain final images.

Figure 1: Qualitative results of our method. ED-NeRF successfully edited 3D scenes with given target text prompts while preserving the original object structure and background regions.

Experimental results

Research questions

• RQ1 Can ED-NeRF accelerate and improve fidelity of text-guided edits for real-world 3D scenes compared to prior NeRF editing methods?
• RQ2 Does editing in the latent space with a refinement layer and DDS-based losses yield better view consistency and region-specific edits?
• RQ3 Can a mask-guided DDS approach effectively constrain edits to target regions while preserving non-target areas?
• RQ4 How does ED-NeRF perform relative to CLIP-based and diffusion-based NeRF editing baselines on qualitative and perceptual metrics?

Key findings

• ED-NeRF achieves faster editing speed and improved output quality versus state-of-the-art 3D editing models.
• A latent-space NeRF with a refinement layer better preserves view consistency and realism in edited views.
• Masked DDS with a reconstruction term yields region-specific edits with reduced undesired deformations.
• Quantitative results show ED-NeRF achieving the highest CLIP Directional Score and strong preservation and view consistency compared to baselines.
• User studies indicate favorable perceptual preferences for ED-NeRF over baseline methods.

Figure 2: Overall pipeline of training and inference stage. (a) We optimize ED-NeRF in the latent space, supervised by source latent. Naively matching NeRF to a latent feature map during optimization can degrade view synthesis quality. (b) Inspired by the embedding process of Stable Diffusion, we in