[Paper Review] GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions

Junjie Wang, Jiemin Fang+3 more|arXiv (Cornell University)|Nov 27, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

GaussianEditor is a novel framework for delicate, text-guided 3D scene editing using 3D Gaussian splatting. It extracts text-based regions of interest (RoI), aligns them to 3D Gaussians via image-space grounding, and applies localized edits with precise control—achieving superior editing quality and 20-minute training speed, outperforming prior methods like Instruct-NeRF2NeRF.

ABSTRACT

Recently, impressive results have been achieved in 3D scene editing with text instructions based on a 2D diffusion model. However, current diffusion models primarily generate images by predicting noise in the latent space, and the editing is usually applied to the whole image, which makes it challenging to perform delicate, especially localized, editing for 3D scenes. Inspired by recent 3D Gaussian splatting, we propose a systematic framework, named GaussianEditor, to edit 3D scenes delicately via 3D Gaussians with text instructions. Benefiting from the explicit property of 3D Gaussians, we design a series of techniques to achieve delicate editing. Specifically, we first extract the region of interest (RoI) corresponding to the text instruction, aligning it to 3D Gaussians. The Gaussian RoI is further used to control the editing process. Our framework can achieve more delicate and precise editing of 3D scenes than previous methods while enjoying much faster training speed, i.e. within 20 minutes on a single V100 GPU, more than twice as fast as Instruct-NeRF2NeRF (45 minutes -- 2 hours).

Motivation & Objective

• To enable fine-grained, localized 3D scene editing guided by natural language instructions.
• To overcome the limitations of 2D diffusion models in localizing editing regions when applied to 3D scenes.
• To leverage the explicit, point-wise nature of 3D Gaussians for precise, individualized editing control.
• To reduce training time compared to existing NeRF-based editing methods while improving editing accuracy.
• To integrate multimodal components (LLM, grounding model, 3D Gaussians) into a unified, interactive editing pipeline.

Proposed method

• Extracts region of interest (RoI) from text instructions using a large language model (LLM) to identify key editing targets.
• Uses a grounding segmentation model to localize the RoI in 2D image space, generating image-level RoI masks.
• Applies a RoI lifting module to project the 2D image RoI back to 3D Gaussian space, minimizing leakage and noise.
• Performs localized editing on 3D Gaussians within the refined RoI using a diffusion model conditioned on the text and spatial constraints.
• Supports interactive refinement via user-provided 3D boxes or point selections to further refine editing regions.
• Integrates scene description generation to improve understanding of spatial relationships, enhancing instruction grounding.

Experimental results

Research questions

• RQ1 Can 3D Gaussian splatting enable more precise and localized 3D scene editing than implicit NeRF-based methods when guided by text?
• RQ2 How can text-based regions of interest be effectively localized in 3D space to avoid global or noisy edits?
• RQ3 Can a multi-stage pipeline combining LLMs, grounding models, and 3D Gaussian manipulation achieve faster and more accurate editing than end-to-end diffusion-based approaches?
• RQ4 What role does scene description generation play in improving the accuracy of spatial understanding for editing tasks?
• RQ5 How does the RoI lifting module reduce editing leakage caused by noisy image-level segmentation?

Key findings

• GaussianEditor achieves significantly more delicate and precise editing than Instruct-NeRF2NeRF, especially in localizing changes to specific object parts.
• The method completes training in under 20 minutes on a single V100 GPU, which is more than twice as fast as Instruct-NeRF2NeRF (45 minutes – 2 hours).
• Abllation studies confirm that all components—Gaussian RoI, Text RoI, and RoI lifting—are essential, with the absence of any leading to significant editing leakage or failure.
• Scene description generation enables accurate understanding of spatial relationships, allowing correct editing of objects like a bench next to a bike, which failed in baseline setups.
• The RoI lifting module effectively reduces noise and leakage from imperfect 2D segmentation, improving 3D editing fidelity.
• User-interactive controls (e.g., 3D boxes, point selection) enable multi-round, fine-grained editing, enhancing usability.