[Paper Review] Geometry-Aware Learning of Maps for Camera Localization

Samarth Brahmbhatt, Jinwei Gu+3 more|arXiv (Cornell University)|Dec 9, 2017

Robotics and Sensor-Based Localization21 references4 citations

TL;DR

This paper proposes MapNet, a deep neural network that learns a data-driven, geometry-aware map representation for camera localization by fusing visual, GPS, and visual odometry inputs. By incorporating geometric constraints—such as relative pose, translation, and rotation—into the loss function during training and inference, MapNet achieves significantly improved localization accuracy over prior DNN-based methods, with further gains through self-supervised adaptation using unlabeled video sequences.

ABSTRACT

Maps are a key component in image-based camera localization and visual SLAM systems: they are used to establish geometric constraints between images, correct drift in relative pose estimation, and relocalize cameras after lost tracking. The exact definitions of maps, however, are often application-specific and hand-crafted for different scenarios (e.g. 3D landmarks, lines, planes, bags of visual words). We propose to represent maps as a deep neural net called MapNet, which enables learning a data-driven map representation. Unlike prior work on learning maps, MapNet exploits cheap and ubiquitous sensory inputs like visual odometry and GPS in addition to images and fuses them together for camera localization. Geometric constraints expressed by these inputs, which have traditionally been used in bundle adjustment or pose-graph optimization, are formulated as loss terms in MapNet training and also used during inference. In addition to directly improving localization accuracy, this allows us to update the MapNet (i.e., maps) in a self-supervised manner using additional unlabeled video sequences from the scene. We also propose a novel parameterization for camera rotation which is better suited for deep-learning based camera pose regression. Experimental results on both the indoor 7-Scenes dataset and the outdoor Oxford RobotCar dataset show significant performance improvement over prior work. The MapNet project webpage is https://goo.gl/mRB3Au.

Motivation & Objective

• Address the inflexibility and application-specific nature of traditional hand-crafted map representations in visual SLAM and image-based localization.
• Enable end-to-end learning of a general-purpose map representation that can adapt to diverse environments and sensor inputs.
• Improve camera localization accuracy by embedding geometric constraints (e.g., relative pose, translation, rotation) directly into the deep learning training process.
• Support continuous, self-supervised map updating using unlabeled video sequences without requiring absolute pose supervision.
• Develop a novel rotation parameterization using the logarithm of unit quaternions to enhance deep learning-based pose regression performance.

Proposed method

• Represent maps as a deep neural network (MapNet) that takes images as input and predicts camera poses, with the map implicitly encoded in the network weights.
• Formulate geometric constraints—such as relative pose from visual odometry, GPS translation, and IMU rotation—into differentiable loss terms during training.
• Introduce a novel rotation parameterization using the logarithm of unit quaternions, which improves optimization stability and regression accuracy in deep networks.
• Enable self-supervised adaptation via MapNet+ by using geometric constraints from unlabeled video sequences to fine-tune the network weights without ground truth poses.
• Fuse MapNet predictions with visual odometry in a moving window using pose graph optimization (PGO) to combine local smoothness with global consistency, resulting in MapNet+PGO.
• Use attention maps to visualize and validate that MapNet focuses on geometrically meaningful image regions, improving temporal consistency.

Experimental results

Research questions

• RQ1 Can a deep neural network be trained to learn a general-purpose, data-driven map representation that generalizes across diverse environments?
• RQ2 How does incorporating geometric constraints (e.g., relative pose, translation, rotation) from auxiliary sensors into the loss function improve camera localization accuracy?
• RQ3 Can MapNet be continuously updated in a self-supervised manner using unlabeled video sequences to improve robustness and adaptability?
• RQ4 Does the proposed logarithmic quaternion parameterization lead to better performance in deep learning-based camera pose regression compared to standard representations?
• RQ5 Can fusing MapNet predictions with visual odometry via PGO further reduce drift and improve localization accuracy in long sequences?

Key findings

• MapNet significantly outperforms prior DNN-based methods like PoseNet on both the indoor 7-Scenes and outdoor Oxford RobotCar datasets, achieving lower translation and rotation errors.
• The integration of geometric constraints into the training loss reduces localization error by leveraging relative pose, translation, and rotation information from visual odometry, GPS, and IMU.
• MapNet+ enables self-supervised adaptation using unlabeled video sequences, allowing the model to improve over time without requiring ground truth poses.
• MapNet+PGO, which fuses MapNet predictions with visual odometry using PGO in a moving window, achieves the best performance, reducing drift and improving accuracy on long sequences.
• The proposed logarithmic quaternion parameterization improves the performance of both PoseNet and MapNet, as shown in quantitative ablation studies on the 7-Scenes dataset.
• Spurious predictions from MapNet+PGO—often caused by overexposed or dynamic regions—can be effectively removed using simple temporal median filtering, as demonstrated in Figure 12.