[Paper Review] Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation

Karl Stelzner, Kristian Kersting+1 more|arXiv (Cornell University)|Apr 2, 2021

3D Shape Modeling and Analysis54 references29 citations

TL;DR

ObSuRF decomposes a single image into multiple volumetric NeRFs, each representing an object, enabling unsupervised 3D object segmentation from RGB-D inputs.

ABSTRACT

We present ObSuRF, a method which turns a single image of a scene into a 3D model represented as a set of Neural Radiance Fields (NeRFs), with each NeRF corresponding to a different object. A single forward pass of an encoder network outputs a set of latent vectors describing the objects in the scene. These vectors are used independently to condition a NeRF decoder, defining the geometry and appearance of each object. We make learning more computationally efficient by deriving a novel loss, which allows training NeRFs on RGB-D inputs without explicit ray marching. After confirming that the model performs equal or better than state of the art on three 2D image segmentation benchmarks, we apply it to two multi-object 3D datasets: A multiview version of CLEVR, and a novel dataset in which scenes are populated by ShapeNet models. We find that after training ObSuRF on RGB-D views of training scenes, it is capable of not only recovering the 3D geometry of a scene depicted in a single input image, but also to segment it into objects, despite receiving no supervision in that regard.

Motivation & Objective

• Motivate learning object-centric 3D representations for downstream tasks (dynamics, reasoning) without supervision.
• Introduce ObSuRF to factor scenes into a set of NeRFs conditioned on latent slots.
• Leverage RGB-D supervision to train NeRFs efficiently without explicit ray marching.
• Provide a principled method to compose multiple NeRFs into a coherent scene function.

Proposed method

• Encode a single RGB image into a set of object slots via a slot-based encoder.
• Condition a shared NeRF decoder for each slot to obtain per-object NeRFs representing geometry and appearance.
• Reframe NeRF rendering as a Poisson-process sufficient formulation to enable depth-based supervision (RGB-D training).
• Compute colors by marginalizing over depth and object components, enabling joint rendering of multi-object scenes.
• Introduce an overlap loss to encourage non-overlapping object volumes while training.
• Render and optimize using RGB-D losses with two NeRF evaluations per pixel, reducing computation compared to standard ray marching.

Experimental results

Research questions

• RQ1 Can a slot-based encoder learn a meaningful decomposition of a scene into independent NeRFs for individual objects without supervision?
• RQ2 How can RGB-D data be utilized to train NeRF-based object decompositions efficiently without explicit ray marching?
• RQ3 Do object-centric NeRF decompositions generalize to new multi-object 3D scenes (e.g., CLEVR-3D, MultiShapeNet) beyond 2D benchmarks?
• RQ4 What impact does enforcing non-overlap between object volumes have on unsupervised 3D segmentation and geometry recovery?

Key findings

• ObSuRF matches or surpasses state-of-the-art 2D unsupervised segmentation benchmarks on CLEVR, dSprites, and Sprite datasets.
• On 3D benchmarks CLEVR-3D and MultiShapeNet, ObSuRF accurately recovers per-object geometry and segmentations from a single RGB view when trained with RGB-D supervision.
• The slot-based, object-centric NeRF decomposition yields lower reconstruction error than a monolithic NeRF autoencoder baseline (NeRF-AE).
• Using RGB-D supervision with a Poisson-process view of ray marching enables two NeRF evaluations per pixel, greatly reducing training cost.
• An overlap loss helps prevent degenerate solutions and improves object separation in 3D scenes (with careful scheduling during training).