[Paper Review] UnsuperPoint: End-to-end Unsupervised Interest Point Detector and Descriptor

Peter Christiansen, Mikkel Fly Kragh+2 more|arXiv (Cornell University)|Jul 9, 2019

Advanced Image and Video Retrieval TechniquesComputer Science47 references73 citations

TL;DR

UnsuperPoint introduces an end-to-end, self-supervised detector and descriptor for interest points that learns point scores, positions, and descriptors without pseudo ground truth, enabling real-time performance and competitive accuracy.

ABSTRACT

It is hard to create consistent ground truth data for interest points in natural images, since interest points are hard to define clearly and consistently for a human annotator. This makes interest point detectors non-trivial to build. In this work, we introduce an unsupervised deep learning-based interest point detector and descriptor. Using a self-supervised approach, we utilize a siamese network and a novel loss function that enables interest point scores and positions to be learned automatically. The resulting interest point detector and descriptor is UnsuperPoint. We use regression of point positions to 1) make UnsuperPoint end-to-end trainable and 2) to incorporate non-maximum suppression in the model. Unlike most trainable detectors, it requires no generation of pseudo ground truth points, no structure-from-motion-generated representations and the model is learned from only one round of training. Furthermore, we introduce a novel loss function to regularize network predictions to be uniformly distributed. UnsuperPoint runs in real-time with 323 frames per second (fps) at a resolution of $224 imes320$ and 90 fps at $480 imes640$. It is comparable or better than state-of-the-art performance when measured for speed, repeatability, localization, matching score and homography estimation on the HPatch dataset.

Motivation & Objective

• Motivate the difficulty of obtaining consistent ground truth for natural image interest points and aim to build an unsupervised detector and descriptor.
• Develop an end-to-end trainable network that jointly predicts point positions, scores, and descriptors.
• Enable learning from a single training pass using self-supervision without SfM or pseudo ground-truth generation.
• Regularize predictions for uniform spatial distribution to improve robustness and coverage.

Proposed method

• Use a shared CNN backbone with task-specific submodules to output point score, position, and descriptor maps.
• Predict relative point positions via regression to enable differentiable training and implicit non-maximum suppression.
• Employ a siamese self-supervised framework with random homographies and augmentations to create corresponding point-pairs across views.
• Introduce a novel UnSupervised Point (USP) loss that aligns point-pair positions and scores and promotes repeatability.
• Add a uniform XY distribution loss to regularize spatial distribution of predicted points.
• Incorporate a descriptor loss with a hinge formulation and a decorrelation loss to learn compact, distinct descriptors.

Experimental results

Research questions

• RQ1 Can an end-to-end network learn both interest point detection and description in a self-supervised manner without pseudo ground truth or SfM guidance?
• RQ2 Does regression-based point localization with integrated NMS-like behavior yield competitive speed and accuracy for real-time applications?
• RQ3 How do auxiliary losses for uniform point distribution and descriptor decorrelation affect repeatability, localization, and matching?
• RQ4 Is interpolation of descriptors within the network beneficial for training and inference?
• RQ5 How does UnsuperPoint perform on standard benchmarks like HPatch compared with state-of-the-art methods?

Key findings

• UnsuperPoint achieves real-time performance (323 fps at 224x320 and 90 fps at 480x640) while delivering competitive or superior speed-related metrics.
• The model learns point positions, scores, and descriptors end-to-end without generating pseudo ground truth points or relying on SfM-based representations.
• Regression-based position prediction enables differentiable training and implicit non-maximum suppression within the network.
• Uniform XY regularization reduces boundary clustering and yields more uniform spatial point distributions.
• Descriptor interpolation, uniform point predictions, and descriptor decorrelation each provide measurable improvements in repeatability, localization accuracy, and matching performance on HPatch.
• The approach demonstrates favorable comparisons to state-of-the-art detectors for speed, repeatability, localization, matching score, and homography estimation on HPatch.