[Paper Review] Full-Frame Scene Coordinate Regression for Image-Based Localization

Xiaotian Li, Juha Ylioinas+1 more|arXiv (Cornell University)|Feb 9, 2018

Robotics and Sensor-Based Localization6 references3 citations

TL;DR

This paper proposes a full-frame scene coordinate regression method using a fully convolutional encoder-decoder network to improve image-based localization robustness and efficiency. By leveraging global context and 2D/3D data augmentation, it achieves state-of-the-art performance, especially on challenging frames with repeated structures, outperforming patch-based methods in accuracy and speed.

ABSTRACT

Image-based localization, or camera relocalization, is a fundamental problem in computer vision and robotics, and it refers to estimating camera pose from an image. Recent state-of-the-art approaches use learning based methods, such as Random Forests (RFs) and Convolutional Neural Networks (CNNs), to regress for each pixel in the image its corresponding position in the scene's world coordinate frame, and solve the final pose via a RANSAC-based optimization scheme using the predicted correspondences. In this paper, instead of in a patch-based manner, we propose to perform the scene coordinate regression in a full-frame manner to make the computation efficient at test time and, more importantly, to add more global context to the regression process to improve the robustness. To do so, we adopt a fully convolutional encoder-decoder neural network architecture which accepts a whole image as input and produces scene coordinate predictions for all pixels in the image. However, using more global context is prone to overfitting. To alleviate this issue, we propose to use data augmentation to generate more data for training. In addition to the data augmentation in 2D image space, we also augment the data in 3D space. We evaluate our approach on the publicly available 7-Scenes dataset, and experiments show that it has better scene coordinate predictions and achieves state-of-the-art results in localization with improved robustness on the hardest frames (e.g., frames with repeated structures).

Motivation & Objective

• Address the limitations of patch-based scene coordinate regression in handling complex scenes with repeated structures.
• Improve localization robustness by incorporating global context into coordinate prediction using a full-image input approach.
• Reduce inference time by enabling efficient full-frame network evaluation compared to patch-wise processing.
• Overcome overfitting in global context learning through novel 2D and 3D data augmentation strategies.
• Achieve state-of-the-art localization accuracy on the 7-Scenes benchmark, particularly on the most challenging frames.

Proposed method

• Employ a fully convolutional encoder-decoder neural network that takes an entire RGB image as input and outputs dense 3D scene coordinates for every pixel.
• Use a two-stage pipeline: first predict full-frame scene coordinates via the proposed Coordinate CNN, then estimate camera pose using RANSAC on predicted correspondences.
• Apply data augmentation in 2D image space (e.g., color jitter, cropping) and in 3D space (e.g., random scene rotations and translations) to regularize the model and prevent overfitting.
• Train the network end-to-end using L2 loss between predicted and ground-truth 3D coordinates for all pixels.
• Optimize for both coordinate prediction accuracy and downstream localization performance via joint training and evaluation.
• Leverage the large receptive field of the full-frame network to better capture scene layout and context, improving generalization on textureless or symmetric scenes.

Experimental results

Research questions

• RQ1 Can full-frame scene coordinate regression with global context improve localization robustness compared to patch-based methods?
• RQ2 How effective is combined 2D and 3D data augmentation in mitigating overfitting when training a full-frame coordinate regression network?
• RQ3 Does the proposed method achieve better performance on the most challenging frames—especially those with repeated structures or low texture?
• RQ4 To what extent does the full-frame architecture reduce inference time compared to patch-based inference?
• RQ5 Can a fully convolutional network with global context produce more accurate intermediate coordinate predictions, leading to improved final localization accuracy?

Key findings

• The proposed full-frame Coordinate CNN achieves 90.3% inlier rate for scene coordinate predictions (vs. 59.4% for DSAC), with a mean error of 28mm, significantly outperforming patch-based baselines.
• Localization performance on the 7-Scenes dataset shows that the method reaches 100% accuracy (cumulative histogram) at lower error thresholds faster than all baselines, especially on the Stairs scene.
• On the hardest frames, the method achieves maximum translational errors below 0.41m and rotational errors below 13°, while DSAC and [25] often suffer from errors exceeding 2–5m and 180°.
• The full-frame network is over 10x faster at inference than the patch-based method (0.02s vs. 0.3s on GTX 1080), and has a smaller model size (~120MB) than the regression-forest-based model (~250MB).
• Without data augmentation, the full-frame network fails to generalize, demonstrating that 2D and 3D augmentation is essential for training robust coordinate prediction.
• The method achieves state-of-the-art localization performance, outperforming both patch-based and keypoint-based methods, particularly in challenging scenarios with repeated structures.