[Paper Review] Neural Volume Rendering: NeRF And Beyond

Frank Dellaert, Yen-Chen Lin|arXiv (Cornell University)|Dec 17, 2020

Computer Graphics and Visualization Techniques39 references33 citations

TL;DR

An annotated bibliography and overview of neural volume rendering with NeRF as the catalyst, surveying pre-NeRF implicit representations, NeRF basics, and a wide range of follow-up directions (speed, dynamics, relighting, shaping, composition, and pose).

ABSTRACT

Besides the COVID-19 pandemic and political upheaval in the US, 2020 was also the year in which neural volume rendering exploded onto the scene, triggered by the impressive NeRF paper by Mildenhall et al. (2020). Both of us have tried to capture this excitement, Frank on a blog post (Dellaert, 2020) and Yen-Chen in a Github collection (Yen-Chen, 2020). This note is an annotated bibliography of the relevant papers, and we posted the associated bibtex file on the repository.

Motivation & Objective

• Define neural volume rendering and its place in neural rendering.
• Summarize key precursors based on implicit surface representations.
• Describe NeRF and why it sparked the field's rapid growth.
• Survey major follow-up directions and their aims (speed, dynamics, relighting, shape priors, composition, pose).
• Provide a curated bibliography linking project sites and arXiv entries for researchers.

Proposed method

• Annotates and contextualizes the evolution from implicit surfaces to neural volume rendering.
• Cites and groups pivotal works (occupancy fields, DeepSDF, PIFu) that underpin implicit functions.
• Explains NeRF's simple MLP + volume integration and its influence (Fourier features, positional encoding).
• Catalogs follow-up efforts across performance, dynamics, relighting, and composition with representative papers.
• Offers an annotated bibliography with links to project sites/arXiv for easy access.

Experimental results

Research questions

• RQ1 What factors limit NeRF in training and rendering speed, and how have subsequent works addressed them?
• RQ2 How can neural radiance fields handle dynamic scenes and non-rigid/deformable content?
• RQ3 What approaches enable relighting and appearance control within neural rendering frameworks?
• RQ4 How can shape priors and latent codes enhance 3D-aware image synthesis and generalization?
• RQ5 What are effective strategies to compose multiple objects into a single neural volume-rendered scene and handle poses?

Key findings

• NeRF sparked a surge of research and an annotated bibliography of NeRF-related work.
• Early work shifted from implicit surfaces/SDFs to volumetric rendering with a simple MLP predicting density and color.
• Follow-up work spans speedups (JaxNeRF, AutoInt), dynamic scenes (NeRFlow, D-NeRF, Nerfies), relighting (NeRV, NeRD, Neural Reflectance Fields, NeRF-W), and scene composition (GIRAFFE, Object-Centric Neural Scene Rendering).
• Researchers explored shape latent codes and pixel-aligned features (pixelNeRF, GRAF, pi-GAN) to improve 3D-aware synthesis and generalization.
• The field shows momentum across performance, dynamics, relighting, and composition, but faces questions about scalability to large scenes and surface-focused representations.