[Paper Review] R2D2: Repeatable and Reliable Detector and Descriptor

Jérôme Revaud, Philippe Weinzaepfel+5 more|arXiv (Cornell University)|Jun 14, 2019

Robotics and Sensor-Based LocalizationEngineering67 references356 citations

TL;DR

R2D2 jointly learns a sparse keypoint detector and local descriptor with a reliability predictor, optimizing for both repeatability and discriminative matching, yielding state-of-the-art results on HPatches and Aachen Day-Night.

ABSTRACT

Interest point detection and local feature description are fundamental steps in many computer vision applications. Classical methods for these tasks are based on a detect-then-describe paradigm where separate handcrafted methods are used to first identify repeatable keypoints and then represent them with a local descriptor. Neural networks trained with metric learning losses have recently caught up with these techniques, focusing on learning repeatable saliency maps for keypoint detection and learning descriptors at the detected keypoint locations. In this work, we argue that salient regions are not necessarily discriminative, and therefore can harm the performance of the description. Furthermore, we claim that descriptors should be learned only in regions for which matching can be performed with high confidence. We thus propose to jointly learn keypoint detection and description together with a predictor of the local descriptor discriminativeness. This allows us to avoid ambiguous areas and leads to reliable keypoint detections and descriptions. Our detection-and-description approach, trained with self-supervision, can simultaneously output sparse, repeatable and reliable keypoints that outperforms state-of-the-art detectors and descriptors on the HPatches dataset. It also establishes a record on the recently released Aachen Day-Night localization dataset.

Motivation & Objective

• Motivate the limitation of using repeatability alone for keypoint detection in learning descriptors.
• Propose a joint detector-descriptor framework that also predicts descriptor discriminativeness (reliability).
• Develop unsupervised losses to promote repeatability and sparsity, and a listwise AP-based loss to optimize descriptor matching.
• Demonstrate the method’s effectiveness on HPatches and visual localization benchmarks (Aachen Day-Night).

Proposed method

• Use a fully convolutional network to output per-pixel descriptors X, a repeatability heatmap S, and a descriptor reliability map R.
• Train S with self-supervised repeatability losses that encourage covariant local maxima under image transformations and local peakiness.
• Train descriptors with a differentiable AP loss, incorporating a reliability-weighted objective so only highly discriminative patches contribute.
• Propose a per-pixel reliability score R to gate descriptor learning, enabling sparse, reliable matches.
• At test time, extract keypoints at local maxima of S and rank them by the product S·R across scales to form the final feature shortlist.
• Provide efficient fully-convolutional implementation to enable dense patch comparison and scalable matching.

Experimental results

Research questions

• RQ1 How can keypoint repeatability and descriptor discriminativeness be learned jointly in a single framework?
• RQ2 Can a reliability predictor for descriptors improve matching by avoiding ambiguous regions?
• RQ3 Do unsupervised repeatability losses produce sparse, uniformly distributed keypoints with high repeatability?
• RQ4 How does the R2D2 approach perform on standard benchmarks (HPatches) and real-world localization benchmarks (Aachen Day-Night)?

Key findings

• The method achieves state-of-the-art results on HPatches, outperforming existing detectors and descriptors in several metrics.
• Jointly learning repeatability and descriptor reliability significantly improves performance versus variants lacking either component.
• The reliability predictor helps suppress regions that are salient but hard to match, improving matching accuracy and localization outcomes.
• On Aachen Day-Night, R2D2 with appropriate configurations matches or surpasses competing methods in visual localization tasks.
• Ablation studies show that separating repeatability and reliability yields better performance than using only a detector or only a descriptor loss.