[Paper Review] Scene Coordinate and Correspondence Learning for Image-Based Localization

Mai Bui, Shadi Albarqouni+2 more|arXiv (Cornell University)|May 22, 2018

Robotics and Sensor-Based LocalizationEngineering26 references16 citations

TL;DR

This paper proposes a deep learning framework for image-based camera re-localization that jointly regresses scene coordinates and their prediction confidences from a single RGB image. By predicting confidence scores for 2D-3D correspondences, the method filters outliers early, enabling more robust pose estimation and refinement, achieving state-of-the-art accuracy of 3.1° rotation and 4.0 cm translation error on the 7-Scenes dataset with RGB input.

ABSTRACT

Scene coordinate regression has become an essential part of current camera re-localization methods. Different versions, such as regression forests and deep learning methods, have been successfully applied to estimate the corresponding camera pose given a single input image. In this work, we propose to regress the scene coordinates pixel-wise for a given RGB image by using deep learning. Compared to the recent methods, which usually employ RANSAC to obtain a robust pose estimate from the established point correspondences, we propose to regress confidences of these correspondences, which allows us to immediately discard erroneous predictions and improve the initial pose estimates. Finally, the resulting confidences can be used to score initial pose hypothesis and aid in pose refinement, offering a generalized solution to solve this task.

Motivation & Objective

• To improve camera re-localization accuracy in challenging conditions such as occlusions and low texture by leveraging deep learning for scene coordinate prediction.
• To address the limitations of RANSAC-based methods by introducing confidence prediction for 2D-3D correspondences to filter erroneous matches early.
• To develop a generalized framework that enhances initial pose hypothesis quality and supports flexible pose refinement without hardcoding optimization thresholds.
• To analyze the impact of different loss functions and regularization techniques on scene coordinate regression quality.
• To demonstrate that confidence prediction enables better performance than direct pose regression or standard correspondence-based methods.

Proposed method

• A deep neural network is trained to predict dense 3D scene coordinates for each pixel in a single RGB input image.
• A novel smoothing regularization term is introduced in the loss function to improve spatial consistency of regressed scene coordinates.
• A separate confidence head is trained to predict the reliability of each 2D-3D correspondence, enabling early rejection of low-confidence matches.
• The confidence scores are used to score and filter pose hypotheses before refinement, improving robustness to outliers.
• Pose estimation is performed using PnP or Kabsch algorithm, with refinement applied using the confidence-weighted correspondences.
• The framework is modular and compatible with various pose estimation algorithms, enabling plug-and-play integration.

Experimental results

Research questions

• RQ1 Can joint regression of scene coordinates and their confidence scores improve camera re-localization accuracy compared to standard correspondence-based methods?
• RQ2 How does the inclusion of a smoothing regularization term in the loss function affect the quality and consistency of regressed scene coordinates?
• RQ3 To what extent can confidence prediction reduce the number of erroneous correspondences and improve the robustness of pose estimation?
• RQ4 How does the proposed method compare to direct pose regression baselines like PoseNet and state-of-the-art correspondence-based methods such as DSAC?
• RQ5 Can the confidence-based filtering mechanism generalize across different scenes and datasets without requiring retraining?

Key findings

• The proposed method achieves a median rotation error of 3.1° and translation error of 4.0 cm on the 7-Scenes dataset using only RGB input, outperforming PoseNet and matching the best state-of-the-art methods.
• With RGB-D input, the method achieves a median error of 2.5° and 5.2 cm, demonstrating significant improvement over RGB-only results and strong performance compared to scene coordinate regression forests.
• The confidence prediction mechanism reduces the number of inlier points needed for pose estimation, enabling accurate results with fewer correspondences.
• The method improves pose accuracy even when using standard PnP and RANSAC, due to the filtering of low-confidence correspondences before hypothesis generation.
• The framework is computationally efficient, running in 1.06 seconds on CPU with GPU-accelerated network inference, outperforming C++-based DSAC in speed despite being implemented in Python.
• The ablation study confirms that confidence prediction significantly enhances pose accuracy, especially in challenging scenes like Stairs, where it reduces error by over 50% compared to baseline methods.