[Paper Review] Unsupervised Discovery of Object Radiance Fields

Hong-Xing Yu, Leonidas Guibas+1 more|arXiv (Cornell University)|Jul 16, 2021

Generative Adversarial Networks and Image SynthesisComputer Science66 references19 citations

TL;DR

This paper proposes unsupervised discovery of Object Radiance Fields (uORF), a method that learns 3D-aware, object-centric scene representations from a single image using only multi-view RGB images as input. By integrating conditional NeRFs with deep inference networks and employing a coarse-to-fine training strategy, uORF achieves state-of-the-art performance in unsupervised 3D scene decomposition, enabling novel tasks like 3D scene editing and generalization to unseen object arrangements and appearances.

ABSTRACT

We study the problem of inferring an object-centric scene representation from a single image, aiming to derive a representation that explains the image formation process, captures the scene's 3D nature, and is learned without supervision. Most existing methods on scene decomposition lack one or more of these characteristics, due to the fundamental challenge in integrating the complex 3D-to-2D image formation process into powerful inference schemes like deep networks. In this paper, we propose unsupervised discovery of Object Radiance Fields (uORF), integrating recent progresses in neural 3D scene representations and rendering with deep inference networks for unsupervised 3D scene decomposition. Trained on multi-view RGB images without annotations, uORF learns to decompose complex scenes with diverse, textured background from a single image. We show that uORF enables novel tasks, such as scene segmentation and editing in 3D, and it performs well on these tasks and on novel view synthesis on three datasets.

Motivation & Objective

• To develop an unsupervised method for learning factorized, 3D-aware, and generative scene representations from a single image.
• To address the challenge of integrating complex 3D-to-2D image formation with deep learning without supervision.
• To enable 3D scene decomposition into objects and background without requiring 3D annotations or category-specific supervision.
• To support novel 3D scene editing and generalization to unseen object arrangements and appearances.
• To improve computational efficiency in neural rendering through progressive coarse-to-fine training.

Proposed method

• uORF models a 3D scene as a composition of individual object radiance fields and a background radiance field using neural implicit representations.
• It employs differentiable volume rendering to render novel views from the inferred radiance fields, with reconstruction losses in pixel space as supervision.
• The method uses a novel progressive coarse-to-fine training schedule to reduce computational cost while improving representation quality.
• It explicitly models background geometry and appearance separately to improve learning on complex scenes with diverse backgrounds.
• Perceptual and adversarial losses are incorporated to enhance image fidelity and robustness in single-image 3D inference.
• The framework leverages conditional NeRFs to enable disentangled inference of object-specific radiance fields from a single input image.

Experimental results

Research questions

• RQ1 Can a deep generative model infer 3D-aware, object-centric scene representations from a single image without any 3D supervision?
• RQ2 Can the method generalize to unseen spatial arrangements of objects, such as densely packed scenes?
• RQ3 Can the model generalize to unseen combinations of object shape and color not seen during training?
• RQ4 How effective is the proposed coarse-to-fine training strategy in balancing computational cost and reconstruction quality?
• RQ5 Can the inferred radiance fields support novel 3D scene editing tasks like object repositioning and background replacement?

Key findings

• uORF achieves a 3D segmentation ARI score of 87.4 ± 0.4 on the Room-Diverse dataset, significantly outperforming baselines.
• On the packed-CLEVR-11 benchmark with 11 closely packed objects, uORF achieves an NV-ARI of 85.0 ± 0.3, demonstrating strong generalization to complex spatial arrangements.
• uORF generalizes to unseen combinations of shape and color, achieving 87.4 ± 0.4 ARI on a test set with only red cylinders and blue spheres—comparable to the oracle model trained on the same distribution.
• The ablation study shows that combining reconstruction, perceptual, and adversarial losses yields the best image quality, reducing LPIPS to 0.1729.
• uORF synthesizes novel views with minimal artifacts and high fidelity, outperforming NeRF-AE and Slot Attention in both quantitative and qualitative evaluations.
• The method enables practical 3D scene editing, such as moving individual objects and modifying backgrounds, due to explicit background modeling and disentangled radiance fields.