[Paper Review] Convolutional networks for real-time 6-DOF camera relocalization.

Alex Kendall, Matthew Koichi Grimes+1 more|arXiv (Cornell University)|May 27, 2015

Robotics and Sensor-Based Localization22 references51 citations

TL;DR

This paper proposes an end-to-end convolutional neural network (CNN) for real-time monocular 6-DOF camera relocalization, achieving 2m and 3° accuracy in large outdoor scenes and 0.5m and 5° indoors by regressing camera pose directly from a single RGB image using transfer learning and a 23-layer deep architecture, with inference at 5ms per frame.

ABSTRACT

Figure 1: Convolutional neural network monocular camera relocalization. Relocalization results for an input image (top), the predicted camera pose of a visual reconstruction (middle), shown again overlaid in red on the original image (bottom). Our system relocalizes to within approximately 2m and 3 ◦ for large outdoor scenes spanning 50, 000m2. We present a robust and real-time monocular six de-gree of freedom relocalization system. Our system trains a convolutional neural network to regress the 6-DOF cam-era pose from a single RGB image in an end-to-end man-ner with no need of additional engineering or graph op-timisation. The algorithm can operate indoors and out-doors in real time, taking 5ms per frame to compute. It obtains approximately 2m and 3◦accuracy for large scale outdoor scenes and 0.5m and 5◦accuracy indoors. This is achieved using an efficient 23 layer deep convnet, demon-strating that convnets can be used to solve complicated out of image plane regression problems. This was made possi-ble by leveraging transfer learning from large scale classi-fication data. We show the convnet localizes from high level features and is robust to difficult lighting, motion blur and different camera intrinsics where point based SIFT registra-tion fails. Furthermore we show how the pose feature that is produced generalizes to other scenes allowing us to regress pose with only a few dozen training examples. 1.

Motivation & Objective

• To enable real-time, robust 6-DOF camera relocalization from a single RGB image without reliance on feature matching or graph optimization.
• To address challenges in monocular relocalization such as lighting variations, motion blur, and differing camera intrinsics where traditional SIFT-based methods fail.
• To demonstrate that deep CNNs can generalize to new scenes with only a few dozen training examples.
• To achieve high accuracy and low inference latency (5ms/frame) for deployment in real-world outdoor and indoor environments.

Proposed method

• A 23-layer deep convolutional neural network is trained end-to-end to regress 6-DOF camera pose (3 translation, 3 rotation) directly from a single RGB image.
• Transfer learning is employed by pre-training the network on large-scale image classification datasets before fine-tuning on the relocalization task.
• The network learns high-level visual features that are robust to lighting changes, motion blur, and varying camera intrinsics.
• Pose regression is performed without explicit geometric constraints or post-processing optimization, enabling real-time inference.
• The model generalizes to new scenes with minimal fine-tuning, requiring only a few dozen training examples per scene.
• Inference is optimized to run in 5ms per frame, enabling real-time operation on standard hardware.

Experimental results

Research questions

• RQ1 Can a deep CNN regress 6-DOF camera pose from a single RGB image with high accuracy and low latency?
• RQ2 How robust is the CNN-based relocalization system to challenging visual conditions such as motion blur and lighting variation?
• RQ3 To what extent can the learned pose features generalize across different scenes with minimal retraining?
• RQ4 Can transfer learning from large-scale classification improve performance on few-shot relocalization in new environments?
• RQ5 How does the end-to-end CNN approach compare to traditional SIFT-based or graph-optimized methods in terms of accuracy and speed?

Key findings

• The system achieves 2m and 3° relocalization accuracy in large outdoor scenes spanning 50,000m².
• Indoors, the system attains 0.5m and 5° accuracy, demonstrating strong performance in complex environments.
• The model computes camera pose in 5ms per frame, enabling real-time operation on standard hardware.
• The network generalizes to new scenes with only a few dozen training examples, reducing data requirements for deployment.
• The approach outperforms point-based SIFT registration under challenging conditions such as motion blur and lighting changes.
• Transfer learning from large-scale classification data significantly improves performance and robustness of the pose regression task.