[Paper Review] InNeRF360: Text-Guided 3D-Consistent Object Inpainting on 360-degree Neural Radiance Fields

Dongqing Wang, Tong Zhang+2 more|arXiv (Cornell University)|May 24, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

InpaintNeRF360 introduces a text-guided, 3D-consistent inpainting framework for unbounded NeRF scenes, leveraging multi-modal prompts from text to guide a promptable segmentation model (SAM) and depth-space warping for view-consistent object removal. It achieves photo-realistic, geometry-aware 3D inpainting by combining 2D image inpainting priors with NeRF fine-tuning using perceptual and L1 losses, enabling accurate, multi-object editing from natural language instructions.

ABSTRACT

We propose InNeRF360, an automatic system that accurately removes text-specified objects from 360-degree Neural Radiance Fields (NeRF). The challenge is to effectively remove objects while inpainting perceptually consistent content for the missing regions, which is particularly demanding for existing NeRF models due to their implicit volumetric representation. Moreover, unbounded scenes are more prone to floater artifacts in the inpainted region than frontal-facing scenes, as the change of object appearance and background across views is more sensitive to inaccurate segmentations and inconsistent inpainting. With a trained NeRF and a text description, our method efficiently removes specified objects and inpaints visually consistent content without artifacts. We apply depth-space warping to enforce consistency across multiview text-encoded segmentations, and then refine the inpainted NeRF model using perceptual priors and 3D diffusion-based geometric priors to ensure visual plausibility. Through extensive experiments in segmentation and inpainting on 360-degree and frontal-facing NeRFs, we show that our approach is effective and enhances NeRF's editability. Project page: https://ivrl.github.io/InNeRF360.

Motivation & Objective

• To address the challenge of 3D scene editing in implicit NeRF representations, particularly object removal with geometric and photometric consistency across 360° views.
• To overcome the limitations of 2D-based inpainting methods that fail on unbounded scenes due to shape variation across viewpoints.
• To enable intuitive, text-based 3D editing without manual segmentation masks, improving user accessibility and control.
• To ensure view consistency and visual plausibility in the inpainted 3D scene through multi-view supervision and perceptual priors.

Proposed method

• Text instructions are encoded into multi-modal prompts to guide the Segment Anything Model (SAM) for multi-view instance segmentation.
• Depth-space warping is applied to propagate and refine 2D segmentation masks across views, enforcing 3D consistency in object identity.
• A pre-trained image inpainter generates perceptually plausible 2D inpainting results on multi-view images using the segmented masks.
• The NeRF model is fine-tuned using a hybrid loss combining pixel-wise L1 loss and masked perceptual loss (LPIPS) to preserve visual coherence.
• View consistency is further enforced by leveraging the 3D consistency of the pre-trained NeRF during fine-tuning.
• The framework supports editing multiple objects or modifying object appearance based on natural language prompts without prompt engineering.

Experimental results

Research questions

• RQ1 Can text-guided 3D inpainting achieve consistent object removal across 360° views in unbounded NeRF scenes?
• RQ2 How effective is depth-space warping in maintaining 3D object identity consistency during multi-view segmentation?
• RQ3 Does combining 2D image inpainting priors with NeRF fine-tuning produce visually plausible and view-consistent 3D results?
• RQ4 How does the method compare to per-frame inpainting in terms of temporal and view consistency?
• RQ5 Can the framework perform accurate, object-level editing without manual mask input, using only natural language instructions?

Key findings

• InpaintNeRF360 achieves 86% user preference for view consistency in the garden scene, outperforming per-frame inpainting (14% preference).
• On the Cup and Starbucks datasets, InpaintNeRF360 achieves LPIPS of 0.6201 and FID of 168.39, outperforming per-frame inpainting (LPIPS: 0.6375, FID: 181.57).
• The inclusion of masked perceptual loss reduces blurry artifacts in the 3D scene, as shown in qualitative comparisons where L1-only training leads to widespread blurriness.
• The method successfully removes multiple objects simultaneously based on a single text instruction, such as 'Remove the flowerpot and flowers'.
• When modified with a mask-conditioned image editor, InpaintNeRF360 enables accurate object stylization, unlike In2n, which applies stylization to large unintended regions.
• The framework generalizes across unbounded and frontal-facing scenes, demonstrating robustness on diverse real-world datasets.