[논문 리뷰] SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis

Guangcong Wang, Zhaoxi Chen외 2명|arXiv (Cornell University)|2023. 03. 28.

Advanced Vision and Imaging인용 수 11

한 줄 요약

SparseNeRF는 로컬 깊이 순위 및 거친 깊이 맵으로부터의 공간 연속성 증류를 통해 소샷 NeRF를 개선하고 LLFF, DTU, 그리고 새로운 NVS-RGBD 데이터셋에서 추론 시간 증가 없이 최첨단 성능을 달성합니다.

ABSTRACT

Neural Radiance Field (NeRF) significantly degrades when only a limited number of views are available. To complement the lack of 3D information, depth-based models, such as DSNeRF and MonoSDF, explicitly assume the availability of accurate depth maps of multiple views. They linearly scale the accurate depth maps as supervision to guide the predicted depth of few-shot NeRFs. However, accurate depth maps are difficult and expensive to capture due to wide-range depth distances in the wild. In this work, we present a new Sparse-view NeRF (SparseNeRF) framework that exploits depth priors from real-world inaccurate observations. The inaccurate depth observations are either from pre-trained depth models or coarse depth maps of consumer-level depth sensors. Since coarse depth maps are not strictly scaled to the ground-truth depth maps, we propose a simple yet effective constraint, a local depth ranking method, on NeRFs such that the expected depth ranking of the NeRF is consistent with that of the coarse depth maps in local patches. To preserve the spatial continuity of the estimated depth of NeRF, we further propose a spatial continuity constraint to encourage the consistency of the expected depth continuity of NeRF with coarse depth maps. Surprisingly, with simple depth ranking constraints, SparseNeRF outperforms all state-of-the-art few-shot NeRF methods (including depth-based models) on standard LLFF and DTU datasets. Moreover, we collect a new dataset NVS-RGBD that contains real-world depth maps from Azure Kinect, ZED 2, and iPhone 13 Pro. Extensive experiments on NVS-RGBD dataset also validate the superiority and generalizability of SparseNeRF. Code and dataset are available at https://sparsenerf.github.io/.

연구 동기 및 목표

• 밀집 다중 뷰 데이터가 이용 불가능할 때 견고한 소샷 신규 시점 합성을 추진한다.
• 사전 학습된 깊이 모델이나 소비자 센서로부터 얻은 거친 깊이 priors를 이용하여 정확한 깊이에 의존하지 않는다.
• 깊이 순위 및 공간 연속성 증류를 도입하여 NeRF 학습을 정규화한다.
• 이 프라이어들이 표준 벤치마크와 새로운 데이터셋에서 기하 및 렌더링 품질을 개선함을 보여준다.

제안 방법

• Base NeRF backbone (Mip-NeRF) trained with color reconstruction loss.
• Depth priors distilled from pre-trained depth models (e.g., DPT) or coarse sensor depths.
• Local depth ranking distillation: enforce NeRF depth ranking matches that of coarse depth within local patches via a ranking loss (Eq. 3).
• Spatial continuity distillation: enforce NeRF depth continuity to mirror local depth continuity from the coarse depth maps (Eq. 4).
• Full objective: L = L_nerf + lambda * R_rank + gamma * R_conti with preset margins and weights.

실험 결과

연구 질문

• RQ1 Can robust depth priors from coarse depth maps improve few-shot NeRF without relying on accurate depth supervision?
• RQ2 Does local depth ranking alone surpass depth scaling when guiding depth in NeRF training?
• RQ3 Does incorporating spatial continuity distillation improve geometric coherence across views?
• RQ4 How do these priors perform on LLFF, DTU, and the new NVS-RGBD dataset, and with different pre-trained depth models?

주요 결과

• SparseNeRF achieves state-of-the-art performance among few-shot NeRF methods on LLFF and DTU in PSNR, SSIM, and LPIPS.
• On LLFF with three views, SparseNeRF attains PSNR 19.86, SSIM 0.624, LPIPS 0.328 (versus 19.08/0.587/0.336 for RegNeRF).
• On DTU with three views, SparseNeRF attains PSNR 19.55, SSIM 0.769, LPIPS 0.201 (vs 18.89/0.745/0.190 for RegNeRF).
• On the new NVS-RGBD dataset, SparseNeRF outperforms RegNeRF, DSNeRF, and MonoSDF across Kinect and ZED 2 sensors (higher PSNR, 0.80+ SSIM, lower LPIPS, and lower depth error).
• Depth ranking distillation and spatial continuity distillation contribute to improved geometry and 3D coherence over baselines (ablation studies show declines without ranking or continuity).
• Using different pre-trained depth models (MiDaS, DPT Hybrid/Large) consistently improves results over baseline, with DPT variants performing best.