[论文解读] NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction

Peng Wang, Lingjie Liu+4 more|arXiv (Cornell University)|Jun 20, 2021

3D Shape Modeling and Analysis被引用 544

一句话总结

NeuS 学习神经隐式距离函数（SDF）并用一种新颖的无偏体积渲染方案进行训练，使在没有前景掩模的情况下实现高保真多视角表面重建。

ABSTRACT

We present a novel neural surface reconstruction method, called NeuS, for reconstructing objects and scenes with high fidelity from 2D image inputs. Existing neural surface reconstruction approaches, such as DVR and IDR, require foreground mask as supervision, easily get trapped in local minima, and therefore struggle with the reconstruction of objects with severe self-occlusion or thin structures. Meanwhile, recent neural methods for novel view synthesis, such as NeRF and its variants, use volume rendering to produce a neural scene representation with robustness of optimization, even for highly complex objects. However, extracting high-quality surfaces from this learned implicit representation is difficult because there are not sufficient surface constraints in the representation. In NeuS, we propose to represent a surface as the zero-level set of a signed distance function (SDF) and develop a new volume rendering method to train a neural SDF representation. We observe that the conventional volume rendering method causes inherent geometric errors (i.e. bias) for surface reconstruction, and therefore propose a new formulation that is free of bias in the first order of approximation, thus leading to more accurate surface reconstruction even without the mask supervision. Experiments on the DTU dataset and the BlendedMVS dataset show that NeuS outperforms the state-of-the-arts in high-quality surface reconstruction, especially for objects and scenes with complex structures and self-occlusion.

研究动机与目标

• 在掩模监督的表面渲染方法之外，推动基于多视图图像的鲁棒表面重建。
• 将表面表示为神经SDF的零等值集，并通过一种新体积渲染公式进行训练。
• 在保持遮挡处理的同时，确保无偏的一阶表面重建。
• 在具有挑战性的数据集（DTU、BlendedMVS）上，无论是否有掩模，均展示出强鲁棒的重建质量。

提出的方法

• 用由多层感知器参数化的神经SDF f 和颜色场 c 来表示场景。
• 引入从逻辑分布推导的S-密度 phi_s，将SDF与体积渲染联系起来。
• 在光线上的权重 w(t) 定义为一个遮挡感知的无偏权重，使用不透明密度 rho 和标准体积渲染框架。
• 提供计算 rho(t) 和 w(t) 的方程（rho(t)=max(-dPhi_s/dt / Phi_s, 0); w(t)=T(t) rho(t) 其中 T(t)=exp(-∫ rho)）。
• 端到端训练，使用颜色损失以及可选的掩模损失和对SDF的Eikonal正则化；采用与NeRF相同的分层采样。
• 在无掩模设置时使用NeRF++处理背景；对几何进行初始化并通过Eikonal及几何先验进行正则化。

实验结果

研究问题

• RQ1 是否可以在不依赖掩模监督的情况下，仅通过体积渲染以2D图像有效训练SDF-based表示并产生高质量表面？
• RQ2 无偏、遮挡感知的体积渲染公式是否相较于传统的基于密度的渲染提升了表面重建的保真度？
• RQ3 在遮挡下，NeuS 在复杂几何和细薄结构上的性能，与 IDR、NeRF 与 UNISURF 相比如何？
• RQ4 Eikonal正则化和几何初始化对学习多视图重建的准确SDF有何影响？

主要发现

• NeuS 在DTU和BlendedMVS上获得卓越的重建质量，在有掩模和无掩模设置下均超越了IDR、NeRF和UNISURF。
• 所提出的无偏且可遮挡感知的权重函数使从神经SDF在没有前景掩模的情况下也能实现准确的零水平表面提取。
• NeuS 能准确重建细薄结构并处理严重自遮挡和突变深度。
• 消融实验表明，简单体积渲染或直接的SDF到体积映射会引入偏差或伪影；正确的权重构建结合Eikonal正则化可提升SDF的准确性。
• 定性与定量结果在具有挑战性的场景中显示出鲁棒性，包括细薄的金属部件和遮挡区域。