[Paper Review] LENS: Localization enhanced by NeRF synthesis

Arthur Moreau, Nathan Piasco+3 more|arXiv (Cornell University)|Oct 13, 2021

Robotics and Sensor-Based LocalizationEngineering33 references17 citations

TL;DR

LENS enhances camera pose regression by generating geometry-consistent, synthetic novel views via NeRF-W, using uniformly sampled virtual camera poses across the scene's 3D geometry. This data augmentation reduces median localization error by 60% on Cambridge Landmarks and 7-Scenes, achieving performance comparable to structure-based methods without architectural changes.

ABSTRACT

Neural Radiance Fields (NeRF) have recently demonstrated photo-realistic results for the task of novel view synthesis. In this paper, we propose to apply novel view synthesis to the robot relocalization problem: we demonstrate improvement of camera pose regression thanks to an additional synthetic dataset rendered by the NeRF class of algorithm. To avoid spawning novel views in irrelevant places we selected virtual camera locations from NeRF internal representation of the 3D geometry of the scene. We further improved localization accuracy of pose regressors using synthesized realistic and geometry consistent images as data augmentation during training. At the time of publication, our approach improved state of the art with a 60% lower error on Cambridge Landmarks and 7-scenes datasets. Hence, the resulting accuracy becomes comparable to structure-based methods, without any architecture modification or domain adaptation constraints. Since our method allows almost infinite generation of training data, we investigated limitations of camera pose regression depending on size and distribution of data used for training on public benchmarks. We concluded that pose regression accuracy is mostly bounded by relatively small and biased datasets rather than capacity of the pose regression model to solve the localization task.

Motivation & Objective

• To address the poor generalization of camera pose regressors caused by biased, non-uniform training data from video sequences.
• To improve pose regression accuracy by generating a diverse, spatially balanced synthetic dataset using NeRF.
• To demonstrate that pose regression can match structure-based method performance when trained on large, well-distributed synthetic data.
• To investigate the impact of data quantity and distribution on pose regression performance, independent of model capacity.
• To enable real-time, low-memory relocalization in embedded robotics systems using synthetic data augmentation.

Proposed method

• LENS uses NeRF-W to train a neural radiance field on real images with known poses, learning 3D scene geometry and appearance.
• Virtual camera poses are generated on a regular 3D grid across the scene, using NeRF’s internal geometry to reject occluded or degenerate views.
• Synthetic images are rendered from these optimized virtual poses, ensuring geometric consistency and photorealism.
• The pose regressor is trained on a combined dataset of real images and NeRF-generated synthetic images to improve generalization.
• A lightweight model, CoordiNet, is used for fast, real-time inference during online localization.
• The method avoids domain shift by mixing real and synthetic data, reducing the gap between synthetic and real image distributions.

Experimental results

Research questions

• RQ1 Can NeRF-based novel view synthesis generate high-quality, geometry-consistent images that improve camera pose regression?
• RQ2 Does training on a uniformly distributed, synthetic dataset across the entire scene significantly reduce localization error compared to real-world video-based training?
• RQ3 To what extent is pose regression accuracy limited by data distribution rather than model capacity?
• RQ4 Can synthetic data augmentation close the performance gap between learning-based and structure-based localization methods?
• RQ5 How does the size and distribution of training data affect the generalization of pose regressors in real-world scenarios?

Key findings

• LENS reduces median translation and rotation error by 60% on the Cambridge Landmarks dataset compared to baseline pose regression.
• On the 7-Scenes Fire scene, median error drops from 27cm/11.7° to 8cm/3.5° when training data is replaced with LENS-generated synthetic views.
• With up to 5000% more training data (synthetic), pose regression achieves 0.03m/1.4° median error on 7-Scenes, matching structure-based method performance.
• The study concludes that pose regression accuracy is primarily limited by data bias and scarcity rather than model capacity or architecture.
• Synthetic-only training fails due to domain gap; mixing real and synthetic data is essential for robustness on real test images.
• LENS enables real-time inference with <50ms per image and <50MB memory usage, making it suitable for embedded robotics applications.