[Paper Review] NeRF--: Neural Radiance Fields Without Known Camera Parameters

Zirui Wang, Shangzhe Wu+3 more|arXiv (Cornell University)|Feb 14, 2021

Advanced Vision and Imaging54 references256 citations

TL;DR

NeRF-- jointly optimizes camera parameters and a NeRF model from forward-facing RGB images, enabling novel view synthesis without COLMAP-style pose supervision, and introduces BLEFF for benchmarking.

ABSTRACT

Considering the problem of novel view synthesis (NVS) from only a set of 2D images, we simplify the training process of Neural Radiance Field (NeRF) on forward-facing scenes by removing the requirement of known or pre-computed camera parameters, including both intrinsics and 6DoF poses. To this end, we propose NeRF$--$, with three contributions: First, we show that the camera parameters can be jointly optimised as learnable parameters with NeRF training, through a photometric reconstruction; Second, to benchmark the camera parameter estimation and the quality of novel view renderings, we introduce a new dataset of path-traced synthetic scenes, termed as Blender Forward-Facing Dataset (BLEFF); Third, we conduct extensive analyses to understand the training behaviours under various camera motions, and show that in most scenarios, the joint optimisation pipeline can recover accurate camera parameters and achieve comparable novel view synthesis quality as those trained with COLMAP pre-computed camera parameters. Our code and data are available at https://nerfmm.active.vision.

Motivation & Objective

• Motivate novel view synthesis when camera intrinsics and poses are unavailable.
• Eliminate the pre-processing COLMAP step by jointly optimizing camera parameters and a NeRF.
• Show that end-to-end photometric reconstruction can recover accurate camera parameters.
• Provide a synthetic BLEFF dataset to benchmark camera estimation and NVS performance.
• Analyze robustness under various camera motions and compare with COLMAP-NeRF pipelines.

Proposed method

• Treat intrinsics and 6DoF poses as learnable parameters and optimize them jointly with NeRF through a photometric reconstruction loss.
• Represent intrinsics with focal length f and assume principal points near image center; represent poses with SE(3) parameters using Rodrigues rotation (axis-angle) for rotation.
• Render pixels by tracing rays using current camera parameters, querying a NeRF MLP FTheta along rays, and integrating radiance via volume rendering.
• Use end-to-end differentiable optimization to update NeRF parameters Theta, camera poses Pi, and focal length f simultaneously.
• Train with randomly sampled pixels per image to compute reconstruction loss against ground-truth RGB images.

Experimental results

Research questions

• RQ1 Can camera intrinsics and 6DoF poses be reliably recovered while training a NeRF from forward-facing images lacking pose supervision?
• RQ2 How does joint optimization compare to two-stage COLMAP-NeRF in terms of NVS quality and camera parameter accuracy under realistic camera motions?
• RQ3 What is the robustness of the joint approach under rotation and translation perturbations in forward-facing scenes?
• RQ4 Does the approach generalize to synthetic BLEFF benchmarks and real-world-like data (RealEstate10K, LLFF) without precomputed camera parameters?

Key findings

• NeRF-- achieves novel view synthesis quality comparable to baseline NeRF that uses COLMAP camera parameters.
• Jointly optimized camera parameters converge toward COLMAP estimates under many conditions, yielding similar NVS performance.
• BLEFF experiments show rotation errors around 5 degrees and focal length errors around 25 pixels on average, with NVS quality close to COLMAP-based NeRF.
• In forward-facing scenes, the joint approach can be more robust to translation perturbations than COLMAP, but may be less robust to large rotations.
• When facing track-to-object or degenerate motions, COLMAP may fail while NeRF-- still provides meaningful renderings in many cases, highlighting complementary strengths.