[Paper Review] Volumetric Disentanglement for 3D Scene Manipulation

Sagie Benaim, Frederik Warburg+2 more|arXiv (Cornell University)|Jun 6, 2022

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes a volumetric disentanglement framework that separates foreground objects from their background in 3D scenes using neural radiance fields, enabling independent manipulation of color, depth, and 3D transformations. By training on 2D masks, views, and poses, the method disentangles foreground and background volumes while preserving illumination, reflections, and occlusions, enabling semantic 3D manipulations such as object camouflage, inpainting, and text-based editing with high consistency across views.

ABSTRACT

Recently, advances in differential volumetric rendering enabled significant breakthroughs in the photo-realistic and fine-detailed reconstruction of complex 3D scenes, which is key for many virtual reality applications. However, in the context of augmented reality, one may also wish to effect semantic manipulations or augmentations of objects within a scene. To this end, we propose a volumetric framework for (i) disentangling or separating, the volumetric representation of a given foreground object from the background, and (ii) semantically manipulating the foreground object, as well as the background. Our framework takes as input a set of 2D masks specifying the desired foreground object for training views, together with the associated 2D views and poses, and produces a foreground-background disentanglement that respects the surrounding illumination, reflections, and partial occlusions, which can be applied to both training and novel views. Our method enables the separate control of pixel color and depth as well as 3D similarity transformations of both the foreground and background objects. We subsequently demonstrate the applicability of our framework on a number of downstream manipulation tasks including object camouflage, non-negative 3D object inpainting, 3D object translation, 3D object inpainting, and 3D text-based object manipulation. Full results are given in our project webpage at https://sagiebenaim.github.io/volumetric-disentanglement/

Motivation & Objective

• To enable semantic 3D scene manipulation in augmented reality by disentangling foreground objects from their background.
• To preserve complex visual effects such as illumination, reflections, and partial occlusions during disentanglement.
• To allow independent control over color, depth, and 3D transformations of both foreground and background objects.
• To support non-negative and depth-preserving manipulations for AR applications with optical-see-through constraints.
• To enable text-based 3D object manipulation using CLIP-guided optimization with semantic consistency.

Proposed method

• The method trains a neural radiance field on the full scene and a second on the background only, using 2D training views, poses, and instance masks.
• Foreground volume is computed via volumetric subtraction: foreground = full scene - background, preserving spatial and radiometric consistency.
• Color and density are disentangled per volume using separate neural networks for each component.
• 3D transformations (translation, rotation, scale) are applied independently to foreground and background volumes using differentiable rendering.
• Text-based manipulation uses CLIP embeddings to guide optimization of foreground object shape and appearance while preserving background semantics.
• The framework supports novel view rendering and maintains consistency across views through differentiable volumetric rendering and implicit scene representation.

Figure 1 : Volumetric disentanglement framework. We introduce a framework for the volumetric disentanglement of foreground objects as well as the background from a full scene (1). Our volumetric disentanglement can then be used for many downstream tasks of interest to designers and artists in AR app

Experimental results

Research questions

• RQ1 Can volumetric disentanglement be achieved in a differentiable, neural radiance field-based framework while preserving complex visual effects like reflections and occlusions?
• RQ2 To what extent can foreground and background objects be manipulated independently in 3D space with respect to color, depth, and pose?
• RQ3 How well does the method generalize to non-negative 3D inpainting and object camouflage under optical-see-through constraints?
• RQ4 Can text-based semantic manipulation of 3D objects be achieved with high fidelity and cross-view consistency?
• RQ5 How robust is the method to noisy or automatically generated 2D masks?

Key findings

• The method successfully disentangles foreground and background volumes while preserving illumination, reflections, and partial occlusions across training and novel views.
• The framework enables consistent 3D manipulations such as object translation, camouflage, and non-negative inpainting with high visual fidelity.
• User studies show that text-based manipulations (e.g., 'strawberry', 'old tree') are semantically consistent and perceptually plausible across multiple views.
• The method handles noisy 2D masks from off-the-shelf segmentation models, demonstrating robustness to imperfect annotations.
• Abllation studies confirm that the volumetric subtraction approach (full - background) outperforms direct foreground field training, which suffers from under-constrained optimization.
• Limitations include incomplete disentanglement of light sources and challenges in manipulating large objects due to insufficient CLIP signal.

Figure 2 : Overview of our disentanglement framework. First, we learn a volumetric representation of the background and full scene using neural radiance fields ( Sec. 3.1 ). Second, by subtracting the full and the background volumes, we obtain a disentangled foreground volume. Third, we can then per