[Paper Review] Learning Feature Descriptors using Camera Pose Supervision

Qianqian Wang, Xiaowei Zhou+2 more|arXiv (Cornell University)|Apr 28, 2020

Robotics and Sensor-Based Localization70 references4 citations

TL;DR

This paper proposes CAPS, a weakly-supervised framework that learns deep feature descriptors using only relative camera poses as supervision, bypassing the need for pixel-level ground-truth correspondences. By leveraging epipolar constraints and a differentiable coarse-to-fine matching layer, CAPS achieves state-of-the-art performance on geometric tasks, outperforming fully supervised methods despite training with only pose supervision.

ABSTRACT

Recent research on learned visual descriptors has shown promising improvements in correspondence estimation, a key component of many 3D vision tasks. However, existing descriptor learning frameworks typically require ground-truth correspondences between feature points for training, which are challenging to acquire at scale. In this paper we propose a novel weakly-supervised framework that can learn feature descriptors solely from relative camera poses between images. To do so, we devise both a new loss function that exploits the epipolar constraint given by camera poses, and a new model architecture that makes the whole pipeline differentiable and efficient. Because we no longer need pixel-level ground-truth correspondences, our framework opens up the possibility of training on much larger and more diverse datasets for better and unbiased descriptors. We call the resulting descriptors CAmera Pose Supervised, or CAPS, descriptors. Though trained with weak supervision, CAPS descriptors outperform even prior fully-supervised descriptors and achieve state-of-the-art performance on a variety of geometric tasks. Project Page: https://qianqianwang68.github.io/CAPS/

Motivation & Objective

• To address the challenge of limited, biased, or sparse ground-truth correspondence annotations in learned feature descriptor training.
• To enable training on larger, more diverse datasets by relaxing supervision to only relative camera poses.
• To develop a differentiable framework that can optimize descriptors using geometric constraints from camera poses rather than pixel-level correspondences.
• To demonstrate that weak supervision via camera poses can yield descriptors matching or exceeding those from fully supervised methods.

Proposed method

• The method formulates a novel epipolar constraint loss that leverages relative camera poses to define geometric relationships between potential correspondences.
• It introduces a differentiable matching layer that enables backpropagation through the correspondence prediction process, making the entire pipeline differentiable.
• A coarse-to-fine architecture computes matches at lower resolution first, then refines them at higher resolution to improve accuracy and efficiency.
• The model uses cycle consistency and uncertainty re-weighting to stabilize training and improve generalization.
• The framework is trained end-to-end using only camera poses, with no need for ground-truth keypoint matches.
• It generates dense descriptors for input images, which can be used with any keypoint detector in downstream tasks.

Experimental results

Research questions

• RQ1 Can feature descriptors be effectively learned using only camera pose supervision, without any pixel-level correspondence annotations?
• RQ2 How can epipolar geometry be leveraged as a weak supervision signal to train deep feature descriptors?
• RQ3 Can a differentiable correspondence module be designed to allow backpropagation through the matching process?
• RQ4 Does the coarse-to-fine architecture improve descriptor quality and training stability compared to single-scale alternatives?
• RQ5 Can weakly-supervised descriptors trained on camera poses match or surpass fully supervised counterparts in performance?

Key findings

• CAPS descriptors outperform prior fully-supervised state-of-the-art methods on multiple geometric tasks, including image matching and SfM, despite being trained without any ground-truth correspondences.
• On the HPatches dataset, CAPS achieves a mean average precision (MMA) of 87.6 and a PCK score of 92.1, surpassing previous fully supervised methods.
• The ablation study shows that the epipolar constraint loss is more effective than triplet or contrastive losses under weak supervision, with cycle consistency providing only marginal gains.
• Training from random initialization (Ours from scratch) still converges successfully, demonstrating the robustness of the proposed loss and architecture.
• The coarse-to-fine module significantly improves performance, suggesting that hierarchical feature representation enhances matching accuracy.
• Even when trained with only camera poses, CAPS achieves competitive SfM results, with a reconstruction completeness of 85.3% and a reprojection error of 1.8 pixels, indicating strong generalization to real-world scenarios.