[Paper Review] Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

Vincent Sitzmann, Michael Zollhöfer+1 more|arXiv (Cornell University)|Jun 4, 2019

Face recognition and analysis679 citations

TL;DR

SRNs propose a continuous, 3D-structure-aware neural scene representation and differentiable renderer that learns geometry and appearance from posed 2D images, enabling novel view synthesis and few-shot reconstruction without 3D supervision.

ABSTRACT

Unsupervised learning with generative models has the potential of discovering rich representations of 3D scenes. While geometric deep learning has explored 3D-structure-aware representations of scene geometry, these models typically require explicit 3D supervision. Emerging neural scene representations can be trained only with posed 2D images, but existing methods ignore the three-dimensional structure of scenes. We propose Scene Representation Networks (SRNs), a continuous, 3D-structure-aware scene representation that encodes both geometry and appearance. SRNs represent scenes as continuous functions that map world coordinates to a feature representation of local scene properties. By formulating the image formation as a differentiable ray-marching algorithm, SRNs can be trained end-to-end from only 2D images and their camera poses, without access to depth or shape. This formulation naturally generalizes across scenes, learning powerful geometry and appearance priors in the process. We demonstrate the potential of SRNs by evaluating them for novel view synthesis, few-shot reconstruction, joint shape and appearance interpolation, and unsupervised discovery of a non-rigid face model.

Motivation & Objective

• Motivate unsupervised discovery of rich 3D scene representations from 2D images without explicit 3D supervision.
• Introduce a continuous scene representation that maps 3D coordinates to a feature vector encoding geometry and appearance.
• Develop a differentiable renderer with neural ray marching to render images from arbitrary viewpoints.
• Demonstrate applications in novel view synthesis, few-shot reconstruction, and discovery of non-rigid face models.

Proposed method

• Represent a scene as a function Phi: R^3 -> R^n that maps world coordinates to a feature vector.
• Use a differentiable neural renderer Theta consisting of a learned ray marcher (RM-LSTM) and a per-pixel MLP color mapper.
• Render by solving for ray-scene intersections via differentiable ray marching and predicting step sizes with the RM-LSTM.
• Allow end-to-end training from posed 2D images without explicit 3D supervision.
• Generalize across scenes by learning a Hypernetwork Psi that maps latent codes z_j to scene parameters phi_j, enabling shared rendering across instances.
• Optimize a joint objective with image loss, depth regularization, and latent-space priors.

Experimental results

Research questions

• RQ1 Can a continuous, 3D-structure-aware scene representation learned from posed 2D images support multi-view consistent novel view synthesis?
• RQ2 To what extent can SRNs learn geometry and appearance priors that generalize across instances within a class?
• RQ3 How well do SRNs perform in few-shot reconstruction and in discovering latent non-rigid deformations (e.g., faces) without 3D supervision?

Key findings

• SRNs achieve multi-view consistent novel view synthesis and outperform several baselines on benchmark tasks.
• SRNs can perform few-shot reconstruction, reconstruct geometry and appearance, and enable latent space interpolation across object instances.
• The model can discover geometry and non-rigid deformations (e.g., face models) in an unsupervised manner solely from posed 2D images.
• Camera pose and intrinsics generalized to unseen configurations, enabling pose extrapolation and close-up renderings.
• A per-pixel color mapper, combined with differentiable ray marching, preserves multi-view consistency without relying on 2D convolutions.