[Paper Review] InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes

Mohamad Shahbazi, Liesbeth Claessens+5 more|arXiv (Cornell University)|Jan 10, 2024

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

InseRF enables text- and 2D bounding box-driven, 3D-consistent generative object insertion in neural 3D scenes without requiring explicit 3D spatial input. It grounds object generation in a single reference view via 2D diffusion editing, lifts the result to 3D using single-view reconstruction, and ensures 3D consistency via depth-guided placement and NeRF fusion, outperforming baselines in localized, realistic object insertion.

ABSTRACT

We introduce InseRF, a novel method for generative object insertion in the NeRF reconstructions of 3D scenes. Based on a user-provided textual description and a 2D bounding box in a reference viewpoint, InseRF generates new objects in 3D scenes. Recently, methods for 3D scene editing have been profoundly transformed, owing to the use of strong priors of text-to-image diffusion models in 3D generative modeling. Existing methods are mostly effective in editing 3D scenes via style and appearance changes or removing existing objects. Generating new objects, however, remains a challenge for such methods, which we address in this study. Specifically, we propose grounding the 3D object insertion to a 2D object insertion in a reference view of the scene. The 2D edit is then lifted to 3D using a single-view object reconstruction method. The reconstructed object is then inserted into the scene, guided by the priors of monocular depth estimation methods. We evaluate our method on various 3D scenes and provide an in-depth analysis of the proposed components. Our experiments with generative insertion of objects in several 3D scenes indicate the effectiveness of our method compared to the existing methods. InseRF is capable of controllable and 3D-consistent object insertion without requiring explicit 3D information as input. Please visit our project page at https://mohamad-shahbazi.github.io/inserf.

Motivation & Objective

• To address the open challenge of 3D-consistent, generative object insertion in complex 3D scenes without explicit 3D spatial guidance.
• To overcome the limitations of existing 3D editing methods that fail at object insertion due to 2D edit inconsistencies and lack of spatial control.
• To enable localized, scene-aware object generation that respects scene appearance, geometry, and spatial context using minimal user input.
• To develop a method that integrates 2D diffusion priors and single-view 3D reconstruction for robust 3D object insertion.

Proposed method

• Ground the 3D object insertion task in a 2D edit within a reference view of the NeRF-reconstructed scene, using a text prompt and 2D bounding box as spatial and semantic guidance.
• Apply a 2D text-to-image diffusion model to generate the target object within the reference view, ensuring semantic and visual alignment with the prompt.
• Lift the 2D-generated object to 3D using a single-view-to-3D reconstruction method, preserving shape and structure from the reference view.
• Estimate depth for the inserted object using monocular depth estimation to guide its 3D placement and ensure geometric consistency across views.
• Fuse the reconstructed object NeRF with the original scene NeRF using a density scaling strategy that accounts for object size and depth to ensure visual harmony.
• Apply an optional refinement step using Instruct-NeRF2NeRF to improve lighting, texture, and scene consistency in the final 3D scene.

Experimental results

Research questions

• RQ1 Can 3D-consistent object insertion be achieved in complex 3D scenes using only a 2D bounding box and text prompt, without explicit 3D spatial input?
• RQ2 How can 2D diffusion models be leveraged to generate 3D objects that are geometrically and visually consistent across multiple views?
• RQ3 What role does depth estimation play in ensuring correct 3D placement of generated objects when only a single 2D reference view is available?
• RQ4 How does the proposed fusion strategy with scene and object NeRF representations affect visual realism and consistency?
• RQ5 To what extent can the method outperform existing 2D-based and 3D-based editing baselines in localized, generative object insertion?

Key findings

• InseRF successfully inserts diverse, 3D-consistent objects into complex 3D scenes using only a text prompt and a single 2D bounding box, achieving localized edits without global scene distortion.
• The method outperforms Instruct-NeRF2NeRF and multi-view inpainting baselines in object insertion, avoiding global scene changes and 3D floaters.
• Scale and radius optimization significantly improves object placement accuracy, resulting in more realistic depth and size alignment with the reference view.
• Object density scaling during NeRF fusion prevents visual artifacts and ensures proper rendering of inserted objects in the scene.
• The optional refinement step enhances texture details and lighting, improving realism and harmonization with the scene.
• Visual ablation confirms that the proposed 2D grounding strategy enables consistent, localized insertion, while baselines fail due to inconsistent 2D edits and lack of 3D control.