[Paper Review] A Comparative Neural Radiance Field (NeRF) 3D Analysis of Camera Poses from HoloLens Trajectories and Structure from Motion

Miriam Jäger, Patrick Hübner+2 more|arXiv (Cornell University)|Apr 20, 2023

Optical measurement and interference techniques4 citations

TL;DR

This paper proposes a direct 3D reconstruction pipeline from Microsoft HoloLens data using Neural Radiance Fields (NeRFs), comparing internal HoloLens trajectory poses with Structure from Motion (SfM)-derived poses. It demonstrates that with pose refinement during NeRF training, internal HoloLens poses achieve PSNR of 27 dB and comparable 3D reconstruction quality to SfM poses, outperforming conventional Multi-View Stereo in detail and completeness on untextured and fine-structured surfaces.

ABSTRACT

Neural Radiance Fields (NeRFs) are trained using a set of camera poses and associated images as input to estimate density and color values for each position. The position-dependent density learning is of particular interest for photogrammetry, enabling 3D reconstruction by querying and filtering the NeRF coordinate system based on the object density. While traditional methods like Structure from Motion are commonly used for camera pose calculation in pre-processing for NeRFs, the HoloLens offers an interesting interface for extracting the required input data directly. We present a workflow for high-resolution 3D reconstructions almost directly from HoloLens data using NeRFs. Thereby, different investigations are considered: Internal camera poses from the HoloLens trajectory via a server application, and external camera poses from Structure from Motion, both with an enhanced variant applied through pose refinement. Results show that the internal camera poses lead to NeRF convergence with a PSNR of 25\,dB with a simple rotation around the x-axis and enable a 3D reconstruction. Pose refinement enables comparable quality compared to external camera poses, resulting in improved training process with a PSNR of 27\,dB and a better 3D reconstruction. Overall, NeRF reconstructions outperform the conventional photogrammetric dense reconstruction using Multi-View Stereo in terms of completeness and level of detail.

Motivation & Objective

• To evaluate the feasibility of using internal HoloLens camera poses—directly extracted via a server application—for training NeRFs in 3D reconstruction.
• To compare NeRF training and reconstruction quality when using HoloLens internal trajectories versus SfM-derived camera poses.
• To assess the impact of pose refinement during NeRF training on convergence and 3D reconstruction quality.
• To benchmark NeRF-based reconstructions against conventional Multi-View Stereo (MVS) in terms of completeness and detail.
• To demonstrate a streamlined, high-resolution 3D reconstruction pipeline directly from HoloLens sensor data using NeRFs.

Proposed method

• Extracted camera poses and RGB images directly from HoloLens trajectories using a custom server application.
• Generated external camera poses via Structure from Motion (SfM) using the same HoloLens images for comparison.
• Trained NeRFs using both internal and SfM camera poses as input, with and without pose refinement during training.
• Applied pose refinement by optimizing extrinsic camera parameters (rotation and translation) during NeRF training.
• Extracted 3D point clouds by sampling NeRF coordinates with high density values, filtering for object surfaces.
• Generated a baseline Multi-View Stereo (MVS) point cloud using the same camera poses for quantitative and qualitative comparison.

Figure 1: Flowchart of the applied investigations. Input data are two different types of camera poses: internal poses of the HoloLens and externally calculated poses via SfM. Subsequently, in each case a pose refinement variant is performed during the training. The four resulting point clouds are ex

Experimental results

Research questions

• RQ1 Can NeRFs trained on internal HoloLens camera poses alone achieve convergence and meaningful 3D reconstruction?
• RQ2 How does the quality of NeRF-based 3D reconstruction compare between internal HoloLens poses and SfM-derived poses?
• RQ3 To what extent does pose refinement during NeRF training improve reconstruction quality when using internal HoloLens poses?
• RQ4 How do NeRF reconstructions compare to conventional Multi-View Stereo (MVS) in terms of completeness and detail on challenging surfaces?
• RQ5 Can a direct pipeline from HoloLens data to high-resolution NeRF-based 3D reconstruction be established without pre-processing via SfM?

Key findings

• Internal HoloLens camera poses, after a simple x-axis rotation, enabled NeRF convergence with a PSNR of 25 dB after approximately 20,000 training epochs.
• Pose refinement during NeRF training improved the PSNR of internal HoloLens-based reconstructions to 27 dB, matching the performance of SfM-derived poses.
• The 3D reconstruction quality from NeRFs using refined internal HoloLens poses was qualitatively comparable to that from SfM poses, with minimal artifacts.
• NeRF-based reconstructions outperformed conventional Multi-View Stereo (MVS) in completeness and level of detail, especially on untextured and fine-structured surfaces like plant branches.
• MVS reconstructions exhibited gray artifacts on fine structures and failed to reconstruct the plant pot, whereas NeRFs preserved these features.
• Color variations in NeRF reconstructions were minor and attributed to directional color modeling, not affecting overall reconstruction quality.

Figure 3: Visualization of an image of the captured Ficus plant as our measurement object using the Microsoft HoloLens RGB camera.