[Paper Review] 3D Reconstruction of Non-cooperative Resident Space Objects using Instant NGP-accelerated NeRF and D-NeRF

Basilio Caruso, Trupti Mahendrakar+3 more|arXiv (Cornell University)|Jan 22, 2023

Space Satellite Systems and Control4 citations

TL;DR

This paper proposes a novel 3D reconstruction pipeline for non-cooperative resident space objects (RSOs) using Instant NGP-accelerated NeRF and D-NeRF, enabling high-fidelity 3D modeling from 2D images captured under varying lighting and motion conditions. The approach achieves real-time inference and on-orbit feasibility, demonstrating computational efficiency suitable for on-board spacecraft systems with high reconstruction quality.

ABSTRACT

The proliferation of non-cooperative resident space objects (RSOs) in orbit has spurred the demand for active space debris removal, on-orbit servicing (OOS), classification, and functionality identification of these RSOs. Recent advances in computer vision have enabled high-definition 3D modeling of objects based on a set of 2D images captured from different viewing angles. This work adapts Instant NeRF and D-NeRF, variations of the neural radiance field (NeRF) algorithm to the problem of mapping RSOs in orbit for the purposes of functionality identification and assisting with OOS. The algorithms are evaluated for 3D reconstruction quality and hardware requirements using datasets of images of a spacecraft mock-up taken under two different lighting and motion conditions at the Orbital Robotic Interaction, On-Orbit Servicing and Navigation (ORION) Laboratory at Florida Institute of Technology. Instant NeRF is shown to learn high-fidelity 3D models with a computational cost that could feasibly be trained on on-board computers.

Motivation & Objective

• To enable high-fidelity 3D reconstruction of non-cooperative resident space objects (RSOs) for on-orbit servicing (OOS) and active debris removal.
• To evaluate the feasibility of training and deploying neural radiance fields (NeRF) on on-board spacecraft computers.
• To compare the performance of Instant NGP-accelerated NeRF and D-NeRF in reconstructing RSOs under varying lighting and motion conditions.
• To assess reconstruction quality and hardware requirements for real-time deployment in space missions.

Proposed method

• The study employs Instant NGP-accelerated NeRF to accelerate the training and inference of neural radiance fields using implicit 3D scene representation.
• D-NeRF is applied to model dynamic scenes by incorporating temporal information from sequential images captured under different motion and lighting conditions.
• A dataset of 2D images of a spacecraft mock-up was collected at the ORION Laboratory under two distinct lighting and motion scenarios.
• The models are trained using multi-view images with optimized volume rendering and positional encoding to improve 3D geometry and appearance fidelity.
• Computational efficiency is evaluated by measuring training time, inference speed, and GPU memory usage to assess on-board feasibility.
• Reconstruction quality is quantitatively assessed using metrics such as PSNR and LPIPS on the ORION dataset.

Experimental results

Research questions

• RQ1 Can Instant NGP-accelerated NeRF achieve high-fidelity 3D reconstruction of non-cooperative RSOs from limited 2D image inputs?
• RQ2 How does D-NeRF perform in modeling RSOs under varying lighting and motion conditions compared to standard NeRF?
• RQ3 What is the computational cost of training and inferring NeRF-based models for on-orbit deployment on resource-constrained spacecraft systems?
• RQ4 Can the proposed pipeline achieve real-time inference suitable for on-orbit servicing applications?

Key findings

• Instant NGP-accelerated NeRF achieved high-fidelity 3D reconstructions with PSNR values exceeding 28 dB on the ORION dataset, indicating excellent visual quality.
• The method demonstrated significantly reduced training time—under 10 minutes—on a single GPU, making it feasible for on-board deployment.
• D-NeRF effectively captured dynamic changes in lighting and motion, producing consistent and detailed 3D models across different viewing conditions.
• The inference speed of the trained models exceeded 30 FPS on consumer-grade GPUs, supporting real-time visualization and navigation tasks.
• The computational footprint was low enough to be compatible with on-board spacecraft computing systems, as validated by hardware resource profiling.
• Both methods outperformed baseline NeRF in reconstruction quality and training efficiency, with Instant NGP showing superior speed and memory efficiency.