[Paper Review] NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

Ben Mildenhall, Pratul P. Srinivasan+4 more|arXiv (Cornell University)|Mar 19, 2020

Advanced Vision and Imaging51 references523 citations

TL;DR

The paper introduces NeRF, a method that represents a scene as a continuous 5D neural radiance field using an MLP and synthesizes novel views via differentiable volume rendering, achieving state-of-the-art results from a sparse set of input images.

ABSTRACT

We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(θ, ϕ)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.

Motivation & Objective

• Motivate a continuous 5D scene representation (x,y,z, theta, phi) that can model complex geometry and view-dependent appearance.
• Represent a scene using a fully-connected neural network to map 5D coordinates to volume density and emitted radiance.
• Develop a differentiable rendering pipeline based on volume rendering to optimize the neural radiance field from RGB images.
• Address efficiency and high-frequency detail via positional encoding and hierarchical sampling to render high-resolution views.

Proposed method

• Represent a scene as a 5D function FΘ(x,y,z,θ,φ) → (c, σ) where c is RGB color and σ is volume density.
• Use an MLP without convolutions to process (x,y,z) into a density σ and a 256-dimensional feature, then combine with viewing direction (θ,φ) to output view-dependent color.
• Render images by casting rays, sampling 3D points along rays, querying the MLP, and applying differentiable volume rendering with a quadrature-based integral.
• Apply stratified sampling along each ray to estimate the color using Ĉ(r) = Σi Ti(1−exp(−σηiδi))ci with Ti as transmittance, enabling differentiability.
• Introduce positional encoding γ(p) that maps inputs to a higher-dimensional space to capture high-frequency content.
• Adopt a two-stage hierarchical sampling with a coarse and a fine network to allocate samples where content is likely visible, improving efficiency.

Experimental results

Research questions

• RQ1 Can a continuous 5D neural radiance field model complex scene geometry and view-dependent appearance from a sparse set of RGB views?
• RQ2 Does differentiable volume rendering of a neural radiance field yield photorealistic novel views that outperform prior neural rendering methods?
• RQ3 Do positional encoding and hierarchical sampling enable stable optimization and high-frequency detail capture in NeRF?
• RQ4 How does NeRF compare to existing neural or voxel-based view synthesis approaches on synthetic and real-world data?

Key findings

• NeRF outperforms prior work on both synthetic and real datasets for novel view synthesis.
• The combination of positional encoding and hierarchical sampling is crucial for high-frequency geometry and appearance reconstruction.
• NeRF requires only RGB images with known poses to optimize, avoiding explicit 3D geometry supervision.
• Compared to baselines, NeRF yields higher fidelity renderings with better multiview consistency and fewer artifacts.
• The approach is capable of rendering high-resolution photorealistic views from relatively sparse input views.