[Paper Review] Fast Image-based Neural Relighting with Translucency-Reflection Modeling

Shizhan Zhu, Shunsuke Saito+4 more|arXiv (Cornell University)|Jun 15, 2023

Computer Graphics and Visualization Techniques4 citations

TL;DR

This paper proposes a volume rendering-based neural relighting framework that end-to-end optimizes geometry and radiance transfer for translucent objects with subsurface scattering (SSS) effects. By learning the radiance transfer gradient from light stage data, it achieves state-of-the-art PSNR performance (5 points higher on average) and high-fidelity relighting across novel lighting and viewpoints without assuming predefined BRDF/BSSRDF models.

ABSTRACT

Image-based lighting (IBL) is a widely used technique that renders objects using a high dynamic range image or environment map. However, aggregating the irradiance at the object's surface is computationally expensive, in particular for non-opaque, translucent materials that require volumetric rendering techniques. In this paper we present a fast neural 3D reconstruction and relighting model that extends volumetric implicit models such as neural radiance fields to be relightable using IBL. It is general enough to handle materials that exhibit complex light transport effects, such as translucency and glossy reflections from detailed surface geometry, producing realistic and compelling results. Rendering can be within a second at 800$ imes$800 resolution (0.72s on an NVIDIA 3090 GPU and 0.30s on an A100 GPU) without engineering optimization. Our code and dataset are available at https://zhusz.github.io/TRHM-Webpage/.

Motivation & Objective

• To address the challenge of relighting translucent and subsurface scattering (SSS) materials in neural rendering, which existing methods fail to handle due to limitations in BRDF/BSSRDF modeling and geometry reconstruction.
• To enable high-fidelity relighting under novel lighting and viewpoints without assuming a specific material representation, such as BRDF or BSSRDF.
• To improve geometric reconstruction quality for optically thick, translucent materials by jointly optimizing appearance and shape via end-to-end learning.
• To collect and release a large-scale, high-fidelity light stage dataset (15TiB) of real objects with prominent SSS effects for future research.

Proposed method

• The method uses volume rendering to estimate the radiance transfer field from images captured under varying lighting conditions in a light stage.
• It formulates the radiance transfer gradient as a learnable field, enabling differentiable optimization of appearance and geometry in a unified framework.
• Appearance cues such as shading, texture, and reflectance are used to refine the geometry end-to-end, improving reconstruction fidelity.
• The approach jointly optimizes geometry and radiance transfer without requiring pre-estimated surfaces or explicit material parameterization.
• It leverages a 512³ resolution density field to extract high-quality meshes via marching cubes, enabling accurate representation of thin, complex structures like ropes.
• The framework is trained on a newly collected light stage dataset featuring rich, high-frequency spatially-varying details across 15TiB of data.

Figure 1 . Our approach reconstructs objects with significant subsurface scattering effects with high fidelity and inserts models into arbitrary environments for relighting. It is fully data-driven and does not assume a particular material representation (such as BRDF or BSSRDF), and can faithfully

Experimental results

Research questions

• RQ1 Can a data-driven neural relighting framework achieve high-fidelity rendering of translucent objects with strong subsurface scattering effects without assuming a predefined BRDF or BSSRDF?
• RQ2 How does end-to-end optimization of geometry and radiance transfer improve reconstruction and relighting quality compared to separated optimization approaches?
• RQ3 To what extent can a volume rendering-based radiance transfer field model handle complex material behaviors such as light transmission beneath surfaces and translucent shadowing?
• RQ4 What performance gains are achievable in PSNR and visual fidelity when using a large-scale, high-fidelity light stage dataset for training and evaluation?

Key findings

• The proposed method achieves a 5-point average PSNR gain over the best existing method across three datasets, demonstrating superior quantitative performance.
• It produces visually plausible and high-fidelity relighting results in both recorded and novel lighting conditions, even for complex translucent materials like soap and candles.
• The method reconstructs high-quality geometry, including thin and intricate structures like ropes, which are challenging for mesh-based representations.
• The end-to-end optimization framework significantly reduces artifacts compared to prior approaches that separate geometry and appearance optimization.
• The method outperforms state-of-the-art baselines such as IRON, InverseTranslucent, and NRTF in both visual fidelity and geometric accuracy on real and synthetic SSS datasets.
• The failure cases highlight limitations in handling specular highlights and sharp shadow boundaries, indicating that specular and shadow modeling are not explicitly captured in the current framework.

Figure 2 . Despite presented with significant subsurface scattering and translucency in the scene, our approach provides the highest geometric reconstruction quality compared to other approaches (NVMC [Hasselgren et al . , 2022 ] ; NRTF [Lyu et al . , 2022 ] via Neus [Wang et al . , 2021 ] ). For ou