[Paper Review] Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters

Axel Barroso-Laguna, Edgar Riba+2 more|arXiv (Cornell University)|Apr 1, 2019

Advanced Image and Video Retrieval Techniques39 references59 citations

TL;DR

Key.Net combines handcrafted image filters with learned CNN filters in a multi-scale, shallow architecture to detect repeatable keypoints with strong performance and efficiency.

ABSTRACT

We introduce a novel approach for keypoint detection task that combines handcrafted and learned CNN filters within a shallow multi-scale architecture. Handcrafted filters provide anchor structures for learned filters, which localize, score and rank repeatable features. Scale-space representation is used within the network to extract keypoints at different levels. We design a loss function to detect robust features that exist across a range of scales and to maximize the repeatability score. Our Key.Net model is trained on data synthetically created from ImageNet and evaluated on HPatches benchmark. Results show that our approach outperforms state-of-the-art detectors in terms of repeatability, matching performance and complexity.

Motivation & Objective

• Motivate robust keypoint detection that remains stable under scale, viewpoint, and illumination changes.
• Propose a hybrid architecture that leverages handcrafted anchors alongside learned filters to reduce parameter count.
• Develop a differentiable multi-scale loss to promote repeatability across scales.
• Introduce a Multi-scale Index Proposal (M-SIP) layer to propose and rank keypoints across scale-space.
• Evaluate on the HPatches benchmark to compare repeatability, matching, and efficiency.

Proposed method

• Use handcrafted filters based on first and second order derivatives (LocalJet) combined with a learnable CNN block.
• Process inputs through a three-level scale-space pyramid with shared weights and upsample/concatenate feature maps.
• Apply a differentiable Index Proposal (IP) layer to extract keypoint coordinates via a spatial softmax within windows.
• Extend IP to Multi-scale Index Proposal (M-SIP) to impose cross-scale robustness by averaging covariant losses across scales.
• Train in a siamese setup with a covariant loss that relates keypoints under known homographies between image pairs.
• Evaluate with a synthetic ImageNet-based training set and HPatches benchmarks, comparing repeatability, IoU, scale range, and matching performance.

Experimental results

Research questions

• RQ1 Can a hybrid detector using both handcrafted and learned filters achieve higher repeatability than fully learned detectors?
• RQ2 Does a multi-scale index proposal loss improve robustness of keypoints across scale variations?
• RQ3 How does Key.Net perform on HPatches in terms of repeatability, IoU, scale stability, and matching when compared to state-of-the-art detectors?
• RQ4 What is the impact of pyramid levels and handcrafted filter usage on model efficiency and accuracy?

Key findings

• Key.Net achieves state-of-the-art or competitive repeatability on HPatches, especially in viewpoint and scale-invariant settings.
• Incorporating handcrafted filters as soft anchors reduces learnable parameters while maintaining performance, enabling 175 FPS inference with 5.7 ms per 600×600 image for Tiny-Key.Net.
• Multi-scale loss (M-SIP) using five window sizes from 8×8 to 40×40 improves repeatability, with the best results when combining all scales.
• Three learnable blocks with handcrafted filters outperform purely learned variants when resources are limited, and performance approaches that of deeper networks as complexity grows.
• Key.Net without scale errors demonstrates strong matching performance when paired with descriptor HardNet, particularly for viewpoint changes.