[Paper Review] Neural Outlier Rejection for Self-Supervised Keypoint Learning

Jiexiong Tang, Hanme Kim+3 more|arXiv (Cornell University)|Dec 23, 2019

Advanced Image and Video Retrieval Techniques31 references4 citations

TL;DR

This paper proposes IO-Net, a self-supervised proxy task using neural outlier rejection to jointly train keypoint detection and description, and introduces KeyPointNet with sub-pixel upsampling and boundary-aware regression. The method achieves state-of-the-art performance in repeatability, localization error, and homography estimation on benchmark datasets without explicit supervision for descriptors.

ABSTRACT

Identifying salient points in images is a crucial component for visual odometry, Structure-from-Motion or SLAM algorithms. Recently, several learned keypoint methods have demonstrated compelling performance on challenging benchmarks. However, generating consistent and accurate training data for interest-point detection in natural images still remains challenging, especially for human annotators. We introduce IO-Net (i.e. InlierOutlierNet), a novel proxy task for the self-supervision of keypoint detection, description and matching. By making the sampling of inlier-outlier sets from point-pair correspondences fully differentiable within the keypoint learning framework, we show that are able to simultaneously self-supervise keypoint description and improve keypoint matching. Second, we introduce KeyPointNet, a keypoint-network architecture that is especially amenable to robust keypoint detection and description. We design the network to allow local keypoint aggregation to avoid artifacts due to spatial discretizations commonly used for this task, and we improve fine-grained keypoint descriptor performance by taking advantage of efficient sub-pixel convolutions to upsample the descriptor feature-maps to a higher operating resolution. Through extensive experiments and ablative analysis, we show that the proposed self-supervised keypoint learning method greatly improves the quality of feature matching and homography estimation on challenging benchmarks over the state-of-the-art.

Motivation & Objective

• Address the challenge of acquiring accurate, consistent training data for keypoint detection in natural images, especially given the difficulty of human annotation for salient points.
• Overcome the limitations of handcrafted features like SIFT and ORB, which are outperformed by learned methods but require costly supervision.
• Develop a self-supervised framework that jointly optimizes keypoint detection and description using a novel proxy task based on inlier-outlier classification.
• Improve descriptor fidelity and matching robustness through architectural innovations in keypoint localization and feature map upsampling.
• Achieve state-of-the-art performance in keypoint matching and homography estimation without relying on explicit keypoint or descriptor supervision.

Proposed method

• Introduce IO-Net, a differentiable proxy task that performs neural outlier rejection on candidate point-pair correspondences to generate supervisory signals for keypoint learning.
• Use the inlier-outlier prediction head to provide gradient signals that guide the keypoint network to learn more discriminative descriptors.
• Design KeyPointNet with a regression head that allows keypoint locations to be predicted outside their grid cells, improving accuracy near and across cell boundaries.
• Apply sub-pixel convolutions to upsample descriptor feature-maps to higher resolution, preserving fine-grained details for improved metric learning.
• Train the keypoint network end-to-end using only the IO-Net loss and a triplet loss for descriptor contrastivity, without explicit keypoint annotations.
• Leverage the flow of gradients from the outlier rejection task to implicitly supervise descriptor learning, even without a direct descriptor loss.

Experimental results

Research questions

• RQ1 Can a self-supervised proxy task based on outlier rejection effectively supervise keypoint detection and description without explicit annotations?
• RQ2 How does differentiable outlier rejection improve the quality of keypoint descriptors and matching performance?
• RQ3 To what extent do architectural improvements like sub-pixel upsampling and boundary-aware regression enhance keypoint detection and descriptor fidelity?
• RQ4 Can a fully self-supervised keypoint network achieve state-of-the-art performance in repeatability, localization error, and homography estimation?
• RQ5 Does the auxiliary IO-Net task provide sufficient supervisory signal to train high-quality descriptors without explicit supervision?

Key findings

• The proposed method achieves 0.686 repeatability at 240×320 resolution, outperforming all prior methods including UnsuperPoint (0.645) and SuperPoint (0.631).
• At 480×640 resolution, the method achieves 0.684 repeatability, significantly outperforming UnsuperPoint (0.612) and SuperPoint (0.593).
• For localization error, the method achieves 0.890 at 240×320 and 0.970 at 480×640, outperforming UnsuperPoint (0.832 and 0.991) at higher resolutions.
• The method achieves 0.544 matching score (M.Score) at 480×640 with 1000 keypoints, surpassing UnsuperPoint (0.383) and SuperPoint (0.281).
• Homography estimation with Cor-5 threshold reaches 0.907 matching score, outperforming all baselines except SIFT in one case, and showing superior robustness in challenging conditions.
• The model runs at 174.5 FPS (with upsampling) and 194.9 FPS (without) on a Titan Xp, demonstrating real-time inference capability.