[Paper Review] Decomposing NeRF for Editing via Feature Field Distillation

Sosuke Kobayashi, Eiichi Matsumoto+1 more|arXiv (Cornell University)|May 31, 2022

Advanced Vision and Imaging103 citations

TL;DR

The paper introduces Distilled Feature Fields (DFFs) that distill 2D image feature encoders into a 3D feature field for NeRFs, enabling zero-shot, query-based semantic decomposition and local edits without retraining the radiance field.

ABSTRACT

Emerging neural radiance fields (NeRF) are a promising scene representation for computer graphics, enabling high-quality 3D reconstruction and novel view synthesis from image observations. However, editing a scene represented by a NeRF is challenging, as the underlying connectionist representations such as MLPs or voxel grids are not object-centric or compositional. In particular, it has been difficult to selectively edit specific regions or objects. In this work, we tackle the problem of semantic scene decomposition of NeRFs to enable query-based local editing of the represented 3D scenes. We propose to distill the knowledge of off-the-shelf, self-supervised 2D image feature extractors such as CLIP-LSeg or DINO into a 3D feature field optimized in parallel to the radiance field. Given a user-specified query of various modalities such as text, an image patch, or a point-and-click selection, 3D feature fields semantically decompose 3D space without the need for re-training and enable us to semantically select and edit regions in the radiance field. Our experiments validate that the distilled feature fields (DFFs) can transfer recent progress in 2D vision and language foundation models to 3D scene representations, enabling convincing 3D segmentation and selective editing of emerging neural graphics representations.

Motivation & Objective

• Enable semantic decomposition of NeRFs for local, object-centric editing without retraining.
• Leverage off-the-shelf 2D feature encoders (e.g., CLIP-LSeg, DINO) as teachers to distill 3D feature fields.
• Support query-based editing via text, image patches, or other modalities.
• Demonstrate improved 3D segmentation and multi-view consistent editing on real-world NeRF scenes.

Proposed method

• Extend NeRF with a 3D feature field f(x) in addition to density sigma(x) and color c(x,d).
• Train f by distilling features rendered along rays to match features from a pre-trained image encoder teacher f_img(I,r) (L_f loss) together with standard photometric loss (L_p).
• Compute 3D segmentation probabilities p(l|x) via dot product between f(x) and a query feature f_q(l) from a zero-shot label space (Eq. 5).
• Enable query-based decomposition to selectively blend or edit regions across multiple NeRFs without retraining, using blending weights derived from segmentation p(l|x).
• Demonstrate interaction modalities including text, image patches, pixel queries, and clustering for region selection, with optional integration with CLIPNeRF for enhanced editing.

Experimental results

Research questions

• RQ1 Can a 3D feature field distilled from 2D pre-trained vision models enable open-set, zero-shot semantic decomposition of NeRF scenes?
• RQ2 Is it possible to edit specific regions in NeRFs via query-based segmentation without retraining the radiance field?
• RQ3 How does distilling 2D features into a 3D field affect novel-view synthesis quality and segmentation performance?
• RQ4 What are the effects of coarse/fine sampling and PE on the quality and smoothness of 3D decomposition and editing?

Key findings

• DFFs enable 3D semantic segmentation of NeRF scenes using text or image queries, achieving competitive mIoU and accuracy on Replica data.
• DFF-based segmentation can outperform a supervised point-cloud model (MinkowskiNet42) in mIoU and accuracy on the evaluated scenes.
• Editability is demonstrated with multi-view-consistent appearance edits, deletions, extractions, and geometric transformations guided by queries.
• Coarse training and removing positional encoding (no-PE) yield smoother volume decomposition and fewer high-frequency artifacts, with trade-offs in representing fine structures.
• Incorporating DFF with CLIPNeRF allows local editing that avoids unintended changes in other scene parts.