[Paper Review] Equivariant Neural Rendering

Emilien Dupont, Miguel Ángel Bautista+5 more|arXiv (Cornell University)|Jun 13, 2020

3D Shape Modeling and Analysis41 references21 citations

TL;DR

This paper introduces a novel framework for learning neural scene representations from 2D images without 3D supervision by enforcing equivariance of the latent representation under 3D transformations. The method enables real-time inference and rendering from a single image, achieving competitive results on both synthetic and complex real-world datasets without requiring pose information at test time or scene alignment during training.

ABSTRACT

We propose a framework for learning neural scene representations directly from images, without 3D supervision. Our key insight is that 3D structure can be imposed by ensuring that the learned representation transforms like a real 3D scene. Specifically, we introduce a loss which enforces equivariance of the scene representation with respect to 3D transformations. Our formulation allows us to infer and render scenes in real time while achieving comparable results to models requiring minutes for inference. In addition, we introduce two challenging new datasets for scene representation and neural rendering, including scenes with complex lighting and backgrounds. Through experiments, we show that our model achieves compelling results on these datasets as well as on standard ShapeNet benchmarks.

Motivation & Objective

• To develop a scene representation learning framework that does not require explicit 3D supervision or scene alignment.
• To enable real-time inference and rendering of novel views from a single image using only relative camera poses during training.
• To introduce a new inductive bias—equivariance under 3D transformations—for learning robust and generalizable 3D scene representations.
• To create and release two new challenging datasets—MugsHQ and 3D Mountains—for evaluating neural rendering on complex, realistic scenes with cluttered backgrounds and complex lighting.
• To demonstrate that equivariance can serve as a strong inductive bias for generalization across diverse scenes without requiring explicit 3D supervision.

Proposed method

• The model enforces equivariance between 3D viewpoint changes and corresponding transformations in the latent scene representation using a differentiable loss function.
• The latent representation is spatial and 3D, structured as a voxel-like grid, allowing for geometric reasoning and 3D-aware feature learning.
• Training uses only posed 2D images and their relative camera poses, without ground-truth 3D annotations or explicit supervision.
• Rendering is performed via a single forward pass through a neural renderer, enabling real-time inference without iterative optimization.
• The method is architecture-agnostic in principle, as the equivariance loss can be applied to any 3D-structured latent representation.
• The model learns to infer a scene representation from a single image and apply transformations (e.g., rotation) to generate novel views.

Experimental results

Research questions

• RQ1 Can equivariance under 3D transformations serve as a strong inductive bias for learning 3D scene representations from 2D images without 3D supervision?
• RQ2 Can a neural scene representation be learned and rendered in real time from a single image without requiring pose information at inference?
• RQ3 How well does the method generalize to complex, real-world scenes with cluttered backgrounds and realistic lighting?
• RQ4 Can the model avoid the need for scene alignment or shared coordinate systems across different scenes during training?
• RQ5 To what extent does equivariance improve generalization across diverse shapes and textures compared to non-equivariant baselines?

Key findings

• The model achieves competitive novel view synthesis performance on both standard ShapeNet benchmarks and the new MugsHQ and 3D Mountains datasets.
• Inference and rendering are performed in real time, with a single forward pass, unlike methods requiring minutes of optimization per image.
• The method generalizes across scenes without requiring alignment or shared coordinate systems, as it relies only on relative camera poses during training.
• The model successfully renders complex visual effects such as reflections and background clutter, demonstrating robustness to realistic scene complexity.
• Failure cases include poor reconstruction of thin structures (e.g., mug handles) and unusual shapes, indicating limitations in modeling fine geometric details.
• The model outperforms non-equivariant baselines in generalization across scenes and maintains consistent performance under viewpoint transformations.