[Paper Review] Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement

Jiaxiang Tang, Hang Zhou+5 more|arXiv (Cornell University)|Mar 3, 2023

3D Shape Modeling and Analysis4 citations

TL;DR

This paper proposes NeRF2Mesh, a framework that recovers high-quality textured surface meshes from multi-view RGB images by adaptively refining a coarse mesh derived from NeRF. It jointly optimizes geometry and appearance through iterative vertex and face density adjustment based on 2D rendering errors, producing real-time renderable meshes with diffuse and view-dependent specular textures.

ABSTRACT

Neural Radiance Fields (NeRF) have constituted a remarkable breakthrough in image-based 3D reconstruction. However, their implicit volumetric representations differ significantly from the widely-adopted polygonal meshes and lack support from common 3D software and hardware, making their rendering and manipulation inefficient. To overcome this limitation, we present a novel framework that generates textured surface meshes from images. Our approach begins by efficiently initializing the geometry and view-dependency decomposed appearance with a NeRF. Subsequently, a coarse mesh is extracted, and an iterative surface refining algorithm is developed to adaptively adjust both vertex positions and face density based on re-projected rendering errors. We jointly refine the appearance with geometry and bake it into texture images for real-time rendering. Extensive experiments demonstrate that our method achieves superior mesh quality and competitive rendering quality.

Motivation & Objective

• Address the representation gap between implicit NeRFs and explicit polygonal meshes, which limits usability in real-time graphics and interactive applications.
• Overcome the inefficiency and incompatibility of NeRF's implicit representation with common 3D software and hardware.
• Enable high-fidelity mesh reconstruction that preserves fine details and view-dependent appearance while maintaining competitive rendering quality.
• Support practical downstream applications by exporting meshes with standard RGB and feature-based textures for use in game engines and CAD tools.

Proposed method

• Initialize geometry and view-dependent appearance using a pre-trained NeRF, decomposing appearance into diffuse and specular components.
• Extract an initial coarse mesh from the NeRF's implicit representation using Marching Cubes or similar methods.
• Apply an iterative adaptive surface refinement algorithm that adjusts vertex positions and face density based on re-projection errors from rendered images.
• Jointly optimize geometry and appearance by minimizing 2D rendering losses, with face refinement guided by error magnitude and surface complexity.
• Bake the optimized diffuse color into an RGB texture and the specular response into a feature texture, enabling real-time rendering via a small MLP in the fragment shader.
• Introduce regularization losses (TV loss, entropy loss) to improve mesh smoothness and prevent overfitting during refinement.

Experimental results

Research questions

• RQ1 Can a coarse mesh extracted from NeRF be effectively refined to achieve high geometric fidelity while preserving view-dependent appearance?
• RQ2 How can adaptive mesh refinement based on 2D rendering errors improve both mesh quality and rendering accuracy compared to fixed or uniform refinement?
• RQ3 To what extent can the resulting textured mesh support real-time rendering and interactive editing in standard 3D graphics pipelines?
• RQ4 What is the trade-off between mesh complexity (vertex/face count) and rendering quality when using adaptive refinement versus uniform refinement?
• RQ5 How does appearance decomposition into diffuse and specular terms affect texture editability and rendering realism?

Key findings

• NeRF2Mesh achieves competitive rendering quality, with a mean PSNR of 29.76 and SSIM of 0.940 on the NeRF-synthetic dataset, outperforming volume-based NeRF in mesh rendering quality.
• On the LLFF dataset, the mesh-based rendering achieves a mean PSNR of 24.75 and SSIM of 0.780, demonstrating strong generalization across scene-level reconstructions.
• On the Mip-NeRF 360 dataset, the method achieves a PSNR of 22.36 and SSIM of 0.493, showing robustness on challenging, complex scenes with background clutter.
• Ablation studies confirm that removing the TV loss reduces mesh smoothness and increases LPIPS (0.483 vs. 0.478), while removing the entropy loss slightly degrades PSNR (22.32 vs. 22.36).
• The mesh size is significantly reduced compared to naive Marching Cubes, with adaptive refinement focusing on high-error regions, resulting in smaller, more efficient meshes.
• The framework successfully supports LODs and real-time rendering, as demonstrated in Figure 10, and enables interactive editing via standard 3D software.