[Paper Review] CMRNet++: Map and camera agnostic monocular visual localization in lidar maps

Daniele Cattaneo, Domenico G. Sorrenti+1 more|arXiv (Cornell University)|Apr 1, 2020

Robotics and Sensor-Based Localization2 citations

TL;DR

CMRNet++ introduces a camera- and map-agnostic monocular visual localization method that decouples metric reasoning from deep learning, enabling accurate localization in unseen environments without retraining. By combining learned feature extraction with geometric calibration, it achieves state-of-the-art performance across KITTI, Argoverse, and Lyft5 datasets, demonstrating robust generalization across diverse cameras and scenes.

ABSTRACT

Localization is a critically essential and crucial enabler of autonomous robots. While deep learning has made significant strides in many computer vision tasks, it is still yet to make a sizeable impact on improving capabilities of metric visual localization. One of the major hindrances has been the inability of existing Convolutional Neural Network (CNN)-based pose regression methods to generalize to previously unseen places. Our recently introduced CMRNet effectively addresses this limitation by enabling map independent monocular localization in LiDAR-maps. In this paper, we now take it a step further by introducing CMRNet++, which is a significantly more robust model that not only generalizes to new places effectively, but is also independent of the camera parameters. We enable this capability by combining deep learning with geometric techniques, and by moving the metric reasoning outside the learning process. In this way, the weights of the network are not tied to a specific camera. Extensive evaluations of CMRNet++ on three challenging autonomous driving datasets, i.e., KITTI, Argoverse, and Lyft5, show that CMRNet++ outperforms CMRNet as well as other baselines by a large margin. More importantly, for the first-time, we demonstrate the ability of a deep learning approach to accurately localize without any retraining or fine-tuning in a completely new environment and independent of the camera parameters.

Motivation & Objective

• Address the lack of generalization in CNN-based monocular visual localization methods to previously unseen environments.
• Overcome the dependency of existing deep learning models on specific camera parameters and LiDAR map structures.
• Enable zero-shot localization in new environments without any retraining or fine-tuning.
• Decouple metric reasoning from the neural network weights to achieve camera independence.
• Demonstrate robust performance across diverse autonomous driving datasets with varying camera setups and map types.

Proposed method

• Introduce a two-stage pipeline: a deep neural network for image feature extraction followed by geometric reasoning for pose estimation.
• Decouple metric reasoning from the network weights by applying geometric calibration post-feature extraction, ensuring camera-agnostic inference.
• Use a learned feature encoder that is invariant to camera intrinsics, trained on diverse camera configurations.
• Apply geometric constraints using LiDDR maps to estimate absolute camera pose from image features without relying on map-specific features.
• Leverage LiDAR maps for metric supervision during training while maintaining camera independence in inference.
• Apply a differentiable geometric layer that maps image features to 3D coordinates in the map, enabling end-to-end training with geometric consistency.

Experimental results

Research questions

• RQ1 Can a deep learning-based monocular localization system achieve accurate localization in completely unseen environments without any retraining?
• RQ2 To what extent can a learned visual localization model generalize across different camera parameters while maintaining high accuracy?
• RQ3 How effective is the decoupling of metric reasoning from neural network weights in improving generalization and camera independence?
• RQ4 Does combining deep learning with geometric reasoning outperform purely learned or purely geometric approaches in real-world autonomous driving scenarios?
• RQ5 Can the model maintain high performance across diverse datasets with varying map structures and camera configurations?

Key findings

• CMRNet++ achieves state-of-the-art performance on KITTI, Argoverse, and Lyft5 datasets, outperforming both CMRNet and other baselines.
• The model demonstrates zero-shot localization in new environments without any retraining or fine-tuning, a first for deep learning-based visual localization.
• CMRNet++ achieves camera-agnostic performance, maintaining high accuracy across different camera intrinsics without model adaptation.
• The decoupling of metric reasoning from the network weights enables robust generalization while preserving high localization accuracy.
• Extensive evaluations confirm that CMRNet++ generalizes effectively across diverse real-world autonomous driving datasets with varying map and camera characteristics.
• The method achieves significant improvements in localization accuracy compared to CMRNet, especially in challenging and unseen environments.