[Paper Review] BeyondPixels: A Comprehensive Review of the Evolution of Neural Radiance Fields

AKM Shahariar Azad Rabby, Chengcui Zhang|arXiv (Cornell University)|Jun 5, 2023

3D Surveying and Cultural Heritage11 citations

TL;DR

This survey consolidates recent NeRF literature, categorizing advances by architecture and scalability, and summarizes techniques to improve quality, efficiency, and large-scale scene handling.

ABSTRACT

Neural rendering combines ideas from classical computer graphics and machine learning to synthesize images from real-world observations. NeRF, short for Neural Radiance Fields, is a recent innovation that uses AI algorithms to create 3D objects from 2D images. By leveraging an interpolation approach, NeRF can produce new 3D reconstructed views of complicated scenes. Rather than directly restoring the whole 3D scene geometry, NeRF generates a volumetric representation called a ``radiance field,'' which is capable of creating color and density for every point within the relevant 3D space. The broad appeal and notoriety of NeRF make it imperative to examine the existing research on the topic comprehensively. While previous surveys on 3D rendering have primarily focused on traditional computer vision-based or deep learning-based approaches, only a handful of them discuss the potential of NeRF. However, such surveys have predominantly focused on NeRF's early contributions and have not explored its full potential. NeRF is a relatively new technique continuously being investigated for its capabilities and limitations. This survey reviews recent advances in NeRF and categorizes them according to their architectural designs, especially in the field of novel view synthesis.

Motivation & Objective

• Motivate and survey NeRF-based research across novel view synthesis and neural rendering.
• Categorize NeRF literature by architectural design and extensions to the original framework.
• Highlight strengths, limitations, and practical challenges in quality, editing, and scalability.
• Propose a new classification scheme to guide future NeRF research directed at overcoming key challenges.

Proposed method

• Explain NeRF fundamentals as a 5D coordinate input to an MLP that outputs density and radiance.
• Review improvements addressing sampling and aliasing via cone tracing and multiscale representations (e.g., Mip-NeRF).
• Summarize point-based, voxel-based, and fusion approaches (e.g., Point-NeRF, NeRFusion) for efficiency and large-scale scenes.
• Describe scalable NeRF variants (FastNeRF, KiloNeRF, Block-NeRF, Mega-NeRF) and their training/inference optimizations.
• Discuss deformation and editability extensions (DRF-Cages, dynamic NeRFs) and their limitations.

Experimental results

Research questions

• RQ1 What are the key architectural categories and extensions that have driven NeRF improvements since its inception?
• RQ2 How do quality, aliasing, and sampling challenges impact NeRF performance, and what techniques mitigate these issues?
• RQ3 What strategies enable NeRF to scale to large-scale scenes and achieve real-time or near-real-time rendering?
• RQ4 What are the current limitations in dynamic/ articulated object representation and scene editing within NeRF frameworks?
• RQ5 How should future NeRF research be organized to address core bottlenecks and applications?

Key findings

• Mip-NeRF reduces aliasing and improves accuracy with a cone-based representation and integrated positional encoding.
• Point-NeRF introduces a point-based representation with pruning/growing to enhance efficiency.
• NeRFusion enables global scene reconstruction by fusing local radiance fields with GRUs and sparse 3D CNNs.
• FastNeRF accelerates rendering to ~200 FPS with a dual-network decomposition and caching, dramatically lowering memory usage.
• KiloNeRF and Block/Mega-NeRF address scalability and large-scale scene handling with tiling, caching, and hierarchical sampling.
• Dynamic/ articulated NeRF approaches (e.g., DeVRF, cage-based deformations) extend NeRF to non-static scenes, albeit with remaining artifacts and limitations.