[Paper Review] Neural Radiance Field-based Visual Rendering: A Comprehensive Review

Mingyuan Yao, Yukang Huo+4 more|arXiv (Cornell University)|Mar 31, 2024

Advanced Vision and Imaging4 citations

TL;DR

This comprehensive review synthesizes recent advances in Neural Radiance Fields (NeRF) for 3D scene reconstruction and novel view synthesis, analyzing core architecture, optimization techniques, datasets, evaluation metrics, and applications across indoor, outdoor, and human body scenes. It provides a structured overview of NeRF variants, performance benchmarks, and future research directions, offering a critical roadmap for researchers in computer vision and graphics.

ABSTRACT

In recent years, Neural Radiance Fields (NeRF) has made remarkable progress in the field of computer vision and graphics, providing strong technical support for solving key tasks including 3D scene understanding, new perspective synthesis, human body reconstruction, robotics, and so on, the attention of academics to this research result is growing. As a revolutionary neural implicit field representation, NeRF has caused a continuous research boom in the academic community. Therefore, the purpose of this review is to provide an in-depth analysis of the research literature on NeRF within the past two years, to provide a comprehensive academic perspective for budding researchers. In this paper, the core architecture of NeRF is first elaborated in detail, followed by a discussion of various improvement strategies for NeRF, and case studies of NeRF in diverse application scenarios, demonstrating its practical utility in different domains. In terms of datasets and evaluation metrics, This paper details the key resources needed for NeRF model training. Finally, this paper provides a prospective discussion on the future development trends and potential challenges of NeRF, aiming to provide research inspiration for researchers in the field and to promote the further development of related technologies.

Motivation & Objective

• To provide a systematic and in-depth analysis of NeRF research literature from the past two years for emerging researchers.
• To clarify the core architecture of NeRF, including its neural network structure, volume rendering process, and loss function.
• To evaluate and compare existing datasets and benchmark metrics used in NeRF model training and performance assessment.
• To classify and analyze NeRF variants based on improvements in rendering quality, inference speed, and generalization across diverse scenes.
• To identify key challenges in NeRF research—such as computational cost, scalability, and complex scene handling—and propose future research directions.

Proposed method

• Conducts a comprehensive survey of NeRF-related literature published between 2022 and 2024, including foundational and recent works.
• Details the original NeRF framework: a multi-layer perceptron (MLP) that implicitly represents 3D radiance fields using 5D coordinates (x, y, z, θ, φ).
• Explains the volume rendering technique used to render novel views by differentiating the radiance field along ray paths.
• Reviews optimization strategies such as hierarchical sampling, positional encoding, and training schedule adjustments to improve convergence and quality.
• Categorizes NeRF variants into groups based on improvements in speed (e.g., Instant-NGP), generalization (e.g., NeRF-So-NeRF), and specialized applications (e.g., FaceCLIPNeRF, GazeNeRF).
• Compares model performance across key metrics: FID, PSNR, SSIM, LPIPS, inference speed, memory usage, and generalization ability on unseen scenes.

Experimental results

Research questions

• RQ1 What are the key architectural components and training mechanisms that define the original NeRF framework?
• RQ2 How have recent NeRF variants improved rendering quality, inference speed, and generalization across diverse scene types?
• RQ3 What are the most widely used datasets and evaluation metrics in NeRF research, and how do they influence model performance assessment?
• RQ4 What are the primary technical and practical challenges limiting NeRF's scalability and real-world deployment?
• RQ5 What future research directions are most promising for overcoming current limitations in NeRF-based 3D scene modeling and rendering?

Key findings

• NeRF has evolved into a foundational technique for neural implicit 3D scene representation, enabling high-fidelity novel view synthesis from sparse input views.
• Techniques like hierarchical sampling and positional encoding significantly improve NeRF's training stability and rendering quality, especially in low-light or texture-poor regions.
• Recent variants such as Instant-NGP and Mip-NeRF360 achieve real-time inference speeds (up to 30 FPS) while maintaining high PSNR (>30) on benchmark datasets.
• Specialized NeRF models like FaceCLIPNeRF and GazeNeRF enable text-conditional 3D facial editing and dynamic expression control, demonstrating strong potential in avatar and VR applications.
• Despite progress, NeRF models still face high computational demands, limited generalization to out-of-distribution scenes, and challenges in handling dynamic or transparent objects.
• The integration of diffusion models (e.g., RODIN) and 3D-aware CLIP-based guidance is emerging as a powerful path toward controllable, high-fidelity 3D generation with improved sample efficiency.