[Paper Review] Depth-supervised NeRF: Fewer Views and Faster Training for Free

Kangle Deng, Andrew Liu+2 more|arXiv (Cornell University)|Jul 6, 2021

Advanced Vision and Imaging38 references45 citations

TL;DR

Depth-supervised NeRF (DS-NeRF) uses sparse depth from structure-from-motion to supervise NeRF, enabling better view synthesis with fewer views and 2–3x faster training.

ABSTRACT

A commonly observed failure mode of Neural Radiance Field (NeRF) is fitting incorrect geometries when given an insufficient number of input views. One potential reason is that standard volumetric rendering does not enforce the constraint that most of a scene's geometry consist of empty space and opaque surfaces. We formalize the above assumption through DS-NeRF (Depth-supervised Neural Radiance Fields), a loss for learning radiance fields that takes advantage of readily-available depth supervision. We leverage the fact that current NeRF pipelines require images with known camera poses that are typically estimated by running structure-from-motion (SFM). Crucially, SFM also produces sparse 3D points that can be used as "free" depth supervision during training: we add a loss to encourage the distribution of a ray's terminating depth matches a given 3D keypoint, incorporating depth uncertainty. DS-NeRF can render better images given fewer training views while training 2-3x faster. Further, we show that our loss is compatible with other recently proposed NeRF methods, demonstrating that depth is a cheap and easily digestible supervisory signal. And finally, we find that DS-NeRF can support other types of depth supervision such as scanned depth sensors and RGB-D reconstruction outputs.

Motivation & Objective

• Motivate NeRF training under sparse views and its tendency to fit incorrect geometry without strong priors.
• Introduce a depth-based supervision signal derived from sparse 3D keypoints produced by SFM (COLMAP).
• Show that depth supervision improves geometry and rendering quality while reducing training time.
• Demonstrate compatibility of the depth supervision loss with other NeRF methods and depth sources (RGB-D and depth sensors).

Proposed method

• Define a depth supervision loss that anchors NeRF ray termination depth to COLMAP keypoints, accounting for depth uncertainty.
• Model the ray termination distribution h(t) as T(t)σ(t) and show it approximates a probability distribution over ray depth.
• Derive a depth loss L_Depth that minimizes KL divergence between the depth-induced distribution and the NeRF ray termination distribution, approximated via a sum over depth samples.
• Combine the color loss L_Color with L_Depth into a joint objective L = L_Color + λ_D L_Depth.
• Show that the depth supervision is complementary and can be plugged into existing NeRF-based methods and RGB-D extensions.

Figure 2 : Few view NeRF. NeRF is susceptible to overfitting when given few training views. As seen by the PSNR gap between train and test renders (left), NeRF has overfit and fails at synthesizing novel views. Further, the depth map (right) and depth error (middle) for NeRF suggest that its density

Experimental results

Research questions

• RQ1 Does depth supervision from sparse COLMAP keypoints improve NeRF performance under few views?
• RQ2 Can DS-NeRF accelerate training while maintaining or improving rendering quality?
• RQ3 Is the depth supervision compatible with other NeRF variants and depth sources (RGB-D, depth sensors)?
• RQ4 How does depth supervision affect depth estimation accuracy on test views?

Key findings

• DS-NeRF renders better images with fewer training views and can train 2–3x faster than vanilla NeRF.
• Depth supervision reduces depth errors on test views, indicating improved geometry recovery.
• KL-divergence-based depth loss often yields fewer artifacts than MSE-based depth supervision.
• DS-NeRF remains effective when using RGB-D or RGB-D-derived depth supervision, enabling denser depth priors.
• DS-NeRF and its variants outperform baselines in few-view settings across NeRF Real, DTU, and Redwood datasets.

Figure 3 : Ray Termination Distribution. (a) We plot various NeRF components over the distance traveled by the ray. Even if a ray traverses through multiple objects (as indicated by the multiple peaks of density $\sigma(t)$ ), we find that the ray termination distribution $h(t)$ is still unimodal. W