[论文解读] Depth-supervised NeRF: Fewer Views and Faster Training for Free

Kangle Deng, Andrew Liu+2 more|arXiv (Cornell University)|Jul 6, 2021

Advanced Vision and Imaging参考文献 38被引用 45

一句话总结

深度监督的 NeRF（DS-NeRF）利用来自结构从运动的稀疏深度来监督 NeRF，从而在更少的视图下实现更好的视图合成，训练速度提高 2–3 倍。

ABSTRACT

A commonly observed failure mode of Neural Radiance Field (NeRF) is fitting incorrect geometries when given an insufficient number of input views. One potential reason is that standard volumetric rendering does not enforce the constraint that most of a scene's geometry consist of empty space and opaque surfaces. We formalize the above assumption through DS-NeRF (Depth-supervised Neural Radiance Fields), a loss for learning radiance fields that takes advantage of readily-available depth supervision. We leverage the fact that current NeRF pipelines require images with known camera poses that are typically estimated by running structure-from-motion (SFM). Crucially, SFM also produces sparse 3D points that can be used as "free" depth supervision during training: we add a loss to encourage the distribution of a ray's terminating depth matches a given 3D keypoint, incorporating depth uncertainty. DS-NeRF can render better images given fewer training views while training 2-3x faster. Further, we show that our loss is compatible with other recently proposed NeRF methods, demonstrating that depth is a cheap and easily digestible supervisory signal. And finally, we find that DS-NeRF can support other types of depth supervision such as scanned depth sensors and RGB-D reconstruction outputs.

研究动机与目标

• 在稀疏视图下激发 NeRF 的训练，并指出在没有强先验时容易拟合错误几何。
• 引入一种基于深度的监督信号，该信号来自 SFM（COLMAP）产生的稀疏 3D 关键点。
• 证明深度监督在提高几何和渲染品质的同时还能减少训练时间。
• 证明深度监督与其他 NeRF 方法及深度源（RGB-D 与深度传感器）兼容。

提出的方法

• 定义一个深度监督损失，将 NeRF 光线终止深度锚定到 COLMAP 关键点，同时考虑深度不确定性。
• 将光线终止分布 h(t) 建模为 T(t)σ(t) 并展示它近似成为一个关于光线深度的概率分布。
• 推导一个深度损失 L_Depth，使其最小化深度诱导分布与 NeRF 光线终止分布之间的 KL 散度，近似通过对深度样本的求和。
• 将颜色损失 L_Color 与 L_Depth 结合成一个联合目标 L = L_Color + λ_D L_Depth。
• 表明深度监督具有互补性，可以接入现有基于 NeRF 的方法和 RGB-D 拓展。

Figure 2 : Few view NeRF. NeRF is susceptible to overfitting when given few training views. As seen by the PSNR gap between train and test renders (left), NeRF has overfit and fails at synthesizing novel views. Further, the depth map (right) and depth error (middle) for NeRF suggest that its density

实验结果

研究问题

• RQ1 来自稀疏 COLMAP 关键点的深度监督在少视图条件下是否提升 NeRF 的性能？
• RQ2 DS-NeRF 能否在保持或提升渲染质量的同时加速训练？
• RQ3 深度监督是否与其他 NeRF 变体和深度来源（RGB-D、深度传感器）兼容？
• RQ4 深度监督对测试视图的深度估计精度有何影响？

主要发现

• DS-NeRF 在更少的训练视图下呈现更好的图像，训练速度比原生 NeRF 快 2–3 倍。
• 深度监督减少测试视图的深度误差，表明几何恢复有所改进。
• KL-散度基的深度损失通常比基于 MSE 的深度监督产生更少的伪影。
• 在使用 RGB-D 或 RGB-D 推导的深度监督时，DS-NeRF 仍然有效，使深度先验更密集。
• 在 NeRF Real、DTU 与 Redwood 数据集的少视图设置中，DS-NeRF及其变体均优于基线。

Figure 3 : Ray Termination Distribution. (a) We plot various NeRF components over the distance traveled by the ray. Even if a ray traverses through multiple objects (as indicated by the multiple peaks of density $\sigma(t)$ ), we find that the ray termination distribution $h(t)$ is still unimodal. W