[Paper Review] Scene Coordinate Regression with Angle-Based Reprojection Loss for Camera Relocalization

Xiaotian Li, Juha Ylioinas+2 more|arXiv (Cornell University)|Aug 15, 2018

Robotics and Sensor-Based Localization40 references3 citations

TL;DR

This paper proposes an angle-based reprojection loss for scene coordinate regression in camera relocalization, enabling end-to-end training without careful initialization and improving pose accuracy. The method leverages multi-view constraints through the new loss, achieving state-of-the-art performance on 7-Scenes and Cambridge Landmarks datasets, with a 8.8 percentage point improvement over DSAC++ on 7-Scenes without initialization.

ABSTRACT

Image-based camera relocalization is an important problem in computer vision and robotics. Recent works utilize convolutional neural networks (CNNs) to regress for pixels in a query image their corresponding 3D world coordinates in the scene. The final pose is then solved via a RANSAC-based optimization scheme using the predicted coordinates. Usually, the CNN is trained with ground truth scene coordinates, but it has also been shown that the network can discover 3D scene geometry automatically by minimizing single-view reprojection loss. However, due to the deficiencies of the reprojection loss, the network needs to be carefully initialized. In this paper, we present a new angle-based reprojection loss, which resolves the issues of the original reprojection loss. With this new loss function, the network can be trained without careful initialization, and the system achieves more accurate results. The new loss also enables us to utilize available multi-view constraints, which further improve performance.

Motivation & Objective

• To eliminate the need for careful initialization in scene coordinate regression networks for camera relocalization.
• To improve localization accuracy by designing a more robust loss function than standard reprojection loss.
• To enable the use of multi-view geometric constraints during training to enhance 3D geometry discovery.
• To demonstrate that the new loss allows convergence and better performance even when ground-truth scene coordinates are unavailable.

Proposed method

• Proposes an angle-based reprojection loss that measures the angular difference between predicted and ground-truth 3D point projections in the image plane, replacing standard L2 reprojection error.
• Uses this loss to train a CNN for predicting 3D scene coordinates from a single RGB image, enabling end-to-end learning without prior coordinate initialization.
• Incorporates multi-view constraints by applying the angle-based loss to corresponding 2D-3D matches across multiple views during training.
• Combines the angle-based loss with photometric reconstruction loss to further regularize predictions and improve geometric consistency.
• Employs a two-stage pipeline: coordinate prediction via CNN, followed by RANSAC-based pose estimation from predicted 2D-3D correspondences.
• Tunes the balance weight λ between multi-view and photometric losses to avoid degradation in performance due to over-regularization.

Experimental results

Research questions

• RQ1 Can a novel loss function eliminate the need for careful initialization in scene coordinate regression for camera relocalization?
• RQ2 Does an angle-based reprojection loss lead to better convergence and higher accuracy than standard reprojection loss?
• RQ3 Can multi-view geometric constraints be effectively leveraged during training with the new loss function?
• RQ4 How does the integration of photometric reconstruction loss affect the performance of the coordinate network?

Key findings

• The proposed angle-based reprojection loss enables successful training of the coordinate network without any initialization, achieving 88.6% accuracy on the 7-Scenes dataset (2cm, 1.0° threshold), a +8.8 percentage point improvement over DSAC++ without initialization.
• On the Cambridge Landmarks dataset, the method achieves better median pose accuracy than DSAC++ on four out of five scenes, demonstrating consistent improvement across diverse environments.
• Incorporating multi-view constraints via the new loss improves performance, with the best results achieved when combining angle-based reprojection loss and photometric reconstruction loss.
• Training with the original reprojection loss without initialization fails to converge in most scenes, highlighting the necessity of the proposed loss for robust training.
• The method achieves 71.8% accuracy on the complete 7-Scenes benchmark (2cm, 1.0°), outperforming DSAC++ without initialization and approaching the performance of DSAC++ with full 3D model supervision (76.1%).
• Sensitivity analysis shows that improper weighting (e.g., λ=50 for photometric loss) degrades performance, indicating the need for careful hyperparameter tuning.