[Paper Review] NeRRF: 3D Reconstruction and View Synthesis for Transparent and Specular Objects with Neural Refractive-Reflective Fields

Xiaoxue Chen, Junchen Liu+3 more|arXiv (Cornell University)|Sep 22, 2023

Advanced Vision and Imaging4 citations

TL;DR

NeRRF proposes a two-stage neural rendering framework for 3D reconstruction and novel view synthesis of transparent and specular objects using only object silhouettes. It leverages differentiable marching tetrahedra with progressive encoding for geometry reconstruction and integrates physically-based ray tracing with Fresnel-based refraction and reflection modeling, achieving state-of-the-art view synthesis quality on both synthetic and real-world datasets with robustness to mask artifacts.

ABSTRACT

Neural radiance fields (NeRF) have revolutionized the field of image-based view synthesis. However, NeRF uses straight rays and fails to deal with complicated light path changes caused by refraction and reflection. This prevents NeRF from successfully synthesizing transparent or specular objects, which are ubiquitous in real-world robotics and A/VR applications. In this paper, we introduce the refractive-reflective field. Taking the object silhouette as input, we first utilize marching tetrahedra with a progressive encoding to reconstruct the geometry of non-Lambertian objects and then model refraction and reflection effects of the object in a unified framework using Fresnel terms. Meanwhile, to achieve efficient and effective anti-aliasing, we propose a virtual cone supersampling technique. We benchmark our method on different shapes, backgrounds and Fresnel terms on both real-world and synthetic datasets. We also qualitatively and quantitatively benchmark the rendering results of various editing applications, including material editing, object replacement/insertion, and environment illumination estimation. Codes and data are publicly available at https://github.com/dawning77/NeRRF.

Motivation & Objective

• To address the challenge of 3D reconstruction and view synthesis for transparent and specular objects, which are difficult to model due to complex light path changes from refraction and reflection.
• To overcome the limitations of standard NeRF, which assumes straight rays and fails on non-Lambertian surfaces.
• To develop a method that reconstructs geometry from only object silhouettes, avoiding reliance on RGB data or depth sensors.
• To enable physically accurate rendering of refraction and reflection using a unified Fresnel-based formulation within a neural radiance field.
• To support downstream applications such as material editing, relighting, and environment illumination estimation.

Proposed method

• Uses Deep Marching Tetrahedra (DMTet) with differentiable rendering to reconstruct 3D geometry from object silhouettes via a progressive encoding strategy.
• Applies a loss based on mask reconstruction and eikonal regularization to optimize the implicit surface representation.
• Introduces a neural refractive-reflective field (NeRRF) that models non-straight ray paths using Snell’s law and the reflection equation.
• Incorporates Fresnel terms to unify the modeling of reflection and refraction based on surface material properties.
• Employs a virtual cone supersampling technique to reduce aliasing and improve rendering smoothness by averaging radiance over a distribution of directions.
• Enables end-to-end training by back-propagating gradients from rendered masks and view synthesis losses.

Experimental results

Research questions

• RQ1 Can a neural radiance field be extended to handle non-straight ray paths caused by refraction and reflection in transparent and specular objects?
• RQ2 How can 3D geometry of non-Lambertian objects be reconstructed from only silhouettes without RGB or depth supervision?
• RQ3 To what extent can a unified Fresnel-based formulation model both reflection and refraction in a single differentiable rendering framework?
• RQ4 How effective is virtual cone supersampling in reducing aliasing artifacts on highly view-dependent surfaces?
• RQ5 Can NeRRF support practical editing tasks such as material editing, relighting, and environment illumination estimation on real-world transparent and specular objects?

Key findings

• NeRRF achieves state-of-the-art novel view synthesis results on both synthetic and real-world datasets, with PSNR improvements of up to 1.8 dB over baseline methods.
• The ablation study shows that progressive encoding reduces chamfer distance by up to 15% compared to non-progressive training, improving surface smoothness.
• Virtual cone supersampling reduces LPIPS by up to 0.025 and increases PSNR by 0.8–1.5 dB across all scenes, significantly reducing surface noise.
• On the real-world dataset, NeRRF successfully reconstructs and renders transparent objects even with imperfect masks from Apple’s cutout tool, demonstrating robustness.
• The method enables high-quality relighting and material editing: e.g., changing a horse from reflective to refractive results in plausible, physically consistent rendering.
• NeRRF achieves accurate environment radiance estimation from reflections alone, as shown in qualitative results where only the ball’s reflection is used to infer the surrounding illumination.