[Paper Review] GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces

Yingwenqi Jiang, Jiadong Tu+5 more|arXiv (Cornell University)|Nov 29, 2023

Computer Graphics and Visualization Techniques4 citations

TL;DR

GaussianShader introduces a novel 3D Gaussian Splatting framework enhanced with a simplified shading function and a geometry-aware normal estimation method to enable high-fidelity, real-time rendering of reflective surfaces. By modeling diffuse color, direct reflections, and residual indirect lighting via spherical harmonics, and using shortest-axis-based normal regularization, the method achieves 1.57dB higher PSNR than Gaussian Splatting on specular datasets while maintaining real-time inference and reducing optimization time to 0.58h—dramatically faster than prior reflective rendering methods.

ABSTRACT

The advent of neural 3D Gaussians has recently brought about a revolution in the field of neural rendering, facilitating the generation of high-quality renderings at real-time speeds. However, the explicit and discrete representation encounters challenges when applied to scenes featuring reflective surfaces. In this paper, we present GaussianShader, a novel method that applies a simplified shading function on 3D Gaussians to enhance the neural rendering in scenes with reflective surfaces while preserving the training and rendering efficiency. The main challenge in applying the shading function lies in the accurate normal estimation on discrete 3D Gaussians. Specifically, we proposed a novel normal estimation framework based on the shortest axis directions of 3D Gaussians with a delicately designed loss to make the consistency between the normals and the geometries of Gaussian spheres. Experiments show that GaussianShader strikes a commendable balance between efficiency and visual quality. Our method surpasses Gaussian Splatting in PSNR on specular object datasets, exhibiting an improvement of 1.57dB. When compared to prior works handling reflective surfaces, such as Ref-NeRF, our optimization time is significantly accelerated (23h vs. 0.58h). Please click on our project website to see more results.

Motivation & Objective

• To address the limitation of 3D Gaussian Splatting in rendering reflective and specular surfaces due to lack of view-dependent appearance modeling.
• To enable efficient, real-time rendering of complex materials including highly reflective surfaces without sacrificing training or inference speed.
• To develop a robust normal estimation framework on discrete 3D Gaussians that aligns with geometric structure for accurate shading computation.
• To integrate a simplified shading model with diffuse, direct, and residual reflection components into the 3D Gaussian representation for improved photorealism.

Proposed method

• Introduces a simplified shading function that models diffuse color, direct reflections, and residual indirect lighting using 3rd-order spherical harmonics.
• Proposes a novel normal representation based on the shortest axis direction of each 3D Gaussian, with a learnable normal residual to improve accuracy.
• Employs a differentiable normal-geometry consistency loss that aligns predicted normals with normals derived from rendered depth maps.
• Uses an explicit environment lighting map for differentiable, view-dependent lighting simulation during end-to-end training.
• Applies a sparse regularization loss to maintain compact 3D Gaussian distributions and avoid overfitting.
• Integrates the shading model into the 3D Gaussian Splatting pipeline, preserving real-time rendering speed and training efficiency.

Experimental results

Research questions

• RQ1 Can a simplified shading function be effectively integrated into 3D Gaussian Splatting to improve rendering quality on reflective surfaces without sacrificing efficiency?
• RQ2 How can accurate surface normals be estimated on discrete, non-continuous 3D Gaussian spheres for reliable shading computation?
• RQ3 What normal representation and regularization strategy best align predicted normals with the underlying geometric structure of 3D Gaussians?
• RQ4 Can explicit lighting representation outperform implicit MLP-based lighting in 3D Gaussian rendering for reflective materials?
• RQ5 To what extent does the residual color term improve rendering quality for complex, view-dependent reflections?

Key findings

• GaussianShader achieves a PSNR of 32.76 on the Tanks and Temples dataset, surpassing Gaussian Splatting (32.05) and Ref-NeRF (31.73), demonstrating superior rendering quality.
• The method reduces optimization time to 0.58 hours, a significant improvement over Ref-NeRF’s 23 hours, while maintaining real-time rendering at 97 FPS.
• The ablation study confirms that the normal-geometry consistency loss and sparse regularization are critical for high-quality normal estimation and rendering.
• The inclusion of residual color (3rd-order SH coefficients) improves PSNR by 0.3dB compared to a model without it, enhancing indirect lighting modeling.
• Visual comparisons show that GaussianShader produces smoother, more accurate normals and more realistic specular highlights than Gaussian Splatting and Ref-NeRF.
• The explicit environment map lighting representation outperforms MLP-based lighting in both PSNR and visual quality, especially on reflective surfaces.