[Paper Review] HF-NeuS: Improved Surface Reconstruction Using High-Frequency Details

Yiqun Wang, Ivan Skorokhodov+1 more|arXiv (Cornell University)|Jun 15, 2022

3D Shape Modeling and Analysis47 citations

TL;DR

HF-NeuS improves neural surface reconstruction by modeling transparency as transformed SDF, using a base plus displacement SDF for high-frequency details, and applying adaptive, spatially varying optimization to focus on surface regions with artifacts.

ABSTRACT

Neural rendering can be used to reconstruct implicit representations of shapes without 3D supervision. However, current neural surface reconstruction methods have difficulty learning high-frequency geometry details, so the reconstructed shapes are often over-smoothed. We develop HF-NeuS, a novel method to improve the quality of surface reconstruction in neural rendering. We follow recent work to model surfaces as signed distance functions (SDFs). First, we offer a derivation to analyze the relationship between the SDF, the volume density, the transparency function, and the weighting function used in the volume rendering equation and propose to model transparency as transformed SDF. Second, we observe that attempting to jointly encode high-frequency and low-frequency components in a single SDF leads to unstable optimization. We propose to decompose the SDF into a base function and a displacement function with a coarse-to-fine strategy to gradually increase the high-frequency details. Finally, we design an adaptive optimization strategy that makes the training process focus on improving those regions near the surface where the SDFs have artifacts. Our qualitative and quantitative results show that our method can reconstruct fine-grained surface details and obtain better surface reconstruction quality than the current state of the art. Code available at https://github.com/yiqun-wang/HFS.

Motivation & Objective

• Motivate better high-frequency surface detail recovery in neural rendering without 3D supervision.
• Derive and adopt a transparency formulation based on transformed signed distance functions (SDF).
• Propose a coarse-to-fine implicit displacement framework to separately learn base and high-frequency surface details.
• Introduce a spatially adaptive strategy to adjust optimization focus around surface regions with artifacts.
• Demonstrate state-of-the-art surface reconstruction on multi-view benchmarks and high-frequency detail scenarios.

Proposed method

• Derive a relationship between the signed distance function (SDF), volume density, transparency, and the weighting function in volume rendering, and propose a monotonic sigmoid-based transparency function T(t)=1/(1+exp(-s f(r(t)))) with s as a trainable slope parameter.
• Decompose the SDF into a base function and an implicit displacement function to separately capture low- and high-frequency geometry, using a coarse-to-fine training scheme and positional encoding for each component.
• Employ a coarse-to-fine strategy with frequency-controlled positional encoding to gradually introduce high-frequency details, and use two MLPs to model the base (fb) and displacement (fd) functions with adaptive sampling.
• Introduce an adaptive, spatially varying scale s for the transparency mapping, computed from the gradient norm of the SDF along the ray and a learned weighting scheme, to focus optimization on surface-adjacent regions with artifacts.
• Train with a radiance loss and an Eikonal regularization loss on both base and detailed SDFs to enforce surface consistency and gradient norm constraints.

Experimental results

Research questions

• RQ1 Can transparency be effectively modeled as a transformed SDF to improve surface reconstruction quality?
• RQ2 Does decomposing the SDF into a base and an implicit displacement function enable better recovery of high-frequency surface details without 3D supervision?
• RQ3 Can a coarse-to-fine, frequency-controlled training scheme stabilize learning of high-frequency geometry in neural surface reconstructions?
• RQ4 Does a spatially adaptive, per-region optimization of the transparency slope s improve reconstruction fidelity near the surface?
• RQ5 How does HF-NeuS compare to NeuS and VolSDF on benchmarks with high-frequency geometry in terms of Chamfer distance and PSNR?

Key findings

• HF-NeuS shows improved surface reconstruction quality over NeuS and VolSDF on high-frequency detail datasets (qualitative and quantitative gains noted).
• A transformed-SDF-based transparency model yields a simpler discretization for density computation and enables inverse CDF sampling, aiding rendering accuracy.
• A base-plus-displacement SDF decomposition with coarse-to-fine frequency control enables the network to learn high-frequency details more effectively than a single SDF.
• Adaptive, spatially varying slope parameter s concentrates optimization near surface regions with artifacts, improving local fidelity.
• Across DTU, NeRF-synthetic, and BlendedMVS datasets, HF-NeuS achieves higher PSNR and lower Chamfer distances, with particularly notable gains on high-frequency-detail scenes.