[Paper Review] Differentiable Rendering: A Survey

Hiroharu Kato, Deniz Beker+5 more|arXiv (Cornell University)|Jun 22, 2020

3D Shape Modeling and Analysis149 references120 citations

TL;DR

The paper provides a comprehensive survey of differentiable rendering (DR) methods across mesh, voxel, point cloud, and neural implicit representations, including neural rendering, applications, evaluation, and open problems.

ABSTRACT

Deep neural networks (DNNs) have shown remarkable performance improvements on vision-related tasks such as object detection or image segmentation. Despite their success, they generally lack the understanding of 3D objects which form the image, as it is not always possible to collect 3D information about the scene or to easily annotate it. Differentiable rendering is a novel field which allows the gradients of 3D objects to be calculated and propagated through images. It also reduces the requirement of 3D data collection and annotation, while enabling higher success rate in various applications. This paper reviews existing literature and discusses the current state of differentiable rendering, its applications and open research problems.

Motivation & Objective

• Motivate differentiable rendering as a bridge between 2D image evidence and 3D scene understanding.
• Summarize the state of DR methods across data representations and their mechanisms.
• Discuss evaluation metrics, libraries, and practical considerations for DR.
• Identify applications and open research problems to guide future work.

Proposed method

• Classify differentiable rendering algorithms by underlying data representations: mesh, voxel, point cloud, and implicit functions.
• Describe analytical and approximated gradient techniques, including global illumination approaches.
• Discuss neural rendering as a learned alternative to handcrafted differentiable renderers.
• Review evaluation methodologies and DR libraries that support research and implementation.
• Outline the pros, limitations, and computational considerations of each representation.
• Address open problems and future directions in differentiable rendering.

Experimental results

Research questions

• RQ1 What are the main differentiable rendering methods for each geometric representation (mesh, voxel, point cloud, implicit) and their differentiation strategies?
• RQ2 How do evaluation methodologies and metrics apply to differentiable rendering methods, and what are the practical limitations?
• RQ3 What are the current applications enabled by differentiable rendering, and what challenges remain for real-time or embedded use?
• RQ4 What open problems and future directions are suggested for differentiable rendering and neural rendering integration?

Key findings

• Mesh-based DR includes analytical derivatives, approximated gradients, and global illumination techniques.
• Voxel-based DR relies on occupancy/TSDF representations and ray marching with differentiable aggregation.
• Point cloud DR handles influence weighting and occlusion, with various strategies for depth and color formation.
• Neural implicit representations enable surface and occupancy modeling with ray-based sampling and differentiable rendering.
• Neural rendering emerges as a data-driven alternative that may improve realism but faces generalization and control challenges.
• The field has no single unified differentiable rendering framework, and practical use requires careful consideration of computation and libraries.