[Paper Review] NeU-NBV: Next Best View Planning Using Uncertainty Estimation in Image-Based Neural Rendering

Liren Jin, Xieyuanli Chen+2 more|arXiv (Cornell University)|Mar 2, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes NeU-NBV, a mapless next best view (NBV) planning framework that uses uncertainty estimation in image-based neural rendering to guide RGB camera deployment for active scene exploration. By selecting views with highest predicted uncertainty, the method efficiently collects informative images without retraining neural radiance fields, leading to superior scene reconstruction quality compared to baselines.

ABSTRACT

Autonomous robotic tasks require actively perceiving the environment to achieve application-specific goals. In this paper, we address the problem of positioning an RGB camera to collect the most informative images to represent an unknown scene, given a limited measurement budget. We propose a novel mapless planning framework to iteratively plan the next best camera view based on collected image measurements. A key aspect of our approach is a new technique for uncertainty estimation in image-based neural rendering, which guides measurement acquisition at the most uncertain view among view candidates, thus maximising the information value during data collection. By incrementally adding new measurements into our image collection, our approach efficiently explores an unknown scene in a mapless manner. We show that our uncertainty estimation is generalisable and valuable for view planning in unknown scenes. Our planning experiments using synthetic and real-world data verify that our uncertainty-guided approach finds informative images leading to more accurate scene representations when compared against baselines.

Motivation & Objective

• To address the challenge of active perception in unknown scenes with limited measurement budgets.
• To develop a mapless NBV planning framework that avoids expensive NeRF retraining during online missions.
• To leverage uncertainty estimation in image-based neural rendering to guide data collection toward the most informative views.
• To improve scene representation quality through targeted image acquisition without explicit global maps.
• To decouple online data collection from offline NeRF training for computational efficiency.

Proposed method

• The framework uses a shared encoder to map nearby reference images into a latent feature space for scene representation.
• It predicts per-pixel uncertainty in novel view synthesis using a learned uncertainty head, without requiring ground truth.
• The next best view is selected as the candidate with the highest average uncertainty across the scene.
• Uncertainty estimation is generalizable across scenes, enabling deployment on new environments without fine-tuning.
• The method incrementally adds new measurements to the image collection, updating uncertainty estimates in real time.
• Scene reconstruction is performed offline using NeRFs trained on images collected via the uncertainty-guided planner.

Figure 1 : Our novel NBV planning framework exploits uncertainty estimation in image-based neural rendering to guide measurement acquisition. Given reference images from the current image collection of the scene (black frustums), our network outputs per-pixel uncertainty estimates at sampled view ca

Experimental results

Research questions

• RQ1 Can uncertainty estimation in image-based neural rendering effectively guide next best view planning in unknown scenes?
• RQ2 How does uncertainty-guided view selection compare to heuristic or random strategies in terms of scene reconstruction quality?
• RQ3 Can the uncertainty estimation generalize to new scenes without retraining or test-time adaptation?
• RQ4 Does collecting images based on uncertainty lead to better NeRF performance with limited data?
• RQ5 Can the framework decouple online data collection from computationally expensive NeRF retraining?

Key findings

• The uncertainty-guided approach achieved higher PSNR (28.35 ± 0.53) and SSIM (0.934 ± 0.004) on the car scene compared to random (25.73 ± 0.83 PSNR, 0.908 ± 0.012 SSIM).
• On the indoor scene, the method achieved 30.46 ± 0.24 PSNR and 0.941 ± 0.003 SSIM, outperforming random planning (28.46 ± 0.92 PSNR, 0.920 ± 0.007 SSIM).
• Qualitative results showed that the method captured more structural details, such as the car bonnet, due to targeted image acquisition.
• The approach reduced reliance on expensive NeRF retraining by decoupling data collection from reconstruction, enabling efficient online operation.
• Images collected via uncertainty-guided planning led to significantly better NeRF training performance with limited data.
• The uncertainty estimation generalised across scenes, enabling effective planning without scene-specific adaptation.

Figure 2 : Overview of our mapless NBV planning framework. We leverage uncertainty estimation in image-based neural rendering to actively guide measurement acquisition in unknown scenes.