[Paper Review] D2-Net: A Trainable CNN for Joint Detection and Description of Local Features

Mihai Dusmanu, Ignacio Rocco+5 more|arXiv (Cornell University)|May 9, 2019

Advanced Image and Video Retrieval Techniques74 citations

TL;DR

D2-Net presents a single trainable CNN that jointly detects keypoints and describes local features by using dense feature maps, achieving state-of-the-art performance on challenging localization benchmarks and competitive results on matching and 3D tasks.

ABSTRACT

In this work we address the problem of finding reliable pixel-level correspondences under difficult imaging conditions. We propose an approach where a single convolutional neural network plays a dual role: It is simultaneously a dense feature descriptor and a feature detector. By postponing the detection to a later stage, the obtained keypoints are more stable than their traditional counterparts based on early detection of low-level structures. We show that this model can be trained using pixel correspondences extracted from readily available large-scale SfM reconstructions, without any further annotations. The proposed method obtains state-of-the-art performance on both the difficult Aachen Day-Night localization dataset and the InLoc indoor localization benchmark, as well as competitive performance on other benchmarks for image matching and 3D reconstruction.

Motivation & Objective

• Motivate robust pixel-level correspondences under challenging imaging conditions (illumination changes, weak texture, motion blur).
• Propose a single CNN that jointly provides dense descriptors and keypoint detections to improve repeatability and matching efficiency.
• Leverage pixel correspondences from large-scale SfM reconstructions to train without extra annotations.
• Demonstrate state-of-the-art localization performance on Aachen Day-Night and InLoc, with competitive image matching and 3D reconstruction results.

Proposed method

• Compute dense feature maps with a CNN that serve as both descriptors and detectors (describe-and-detect).
• Descriptors are the n-channel feature vectors at each pixel, L2-normalized for matching.
• Detections are local maxima across feature map channels with soft, differentiable channel selection and a soft local-maximum score.
• Multiscale detection is achieved via image pyramids with scale fusion and response gating to avoid re-detection across scales.
• Training uses an extended triplet margin ranking loss that jointly optimizes descriptor distinctiveness and keypoint repeatability, weighted by soft detection scores.
• During testing, reduce memory/compute via architectural adjustments (replace last pooling, apply dilated convolutions, refine keypoint positions).

Experimental results

Research questions

• RQ1 Can a single CNN serving as both detector and descriptor produce robust, repeatable keypoints under challenging illumination and viewpoint changes?
• RQ2 How does postponing detection to the descriptor feature maps affect matching accuracy and localization performance compared to traditional detect-then-describe pipelines?
• RQ3 Can dense, trainable descriptors with joint detection achieve competitive or superior performance on standard benchmarks (Aachen Day-Night, InLoc, HPatches) while remaining practical for SfM/localization pipelines?

Key findings

• Achieves best overall performance for image matching at thresholds of 6.5 pixels or more on the HPatches-based evaluation, outperforming several detect-then-describe baselines.
• On challenging localization tasks, D2-Net attains state-of-the-art results on Aachen Day-Night and strong performance on InLoc, often surpassing dense and sparse baselines in pose accuracy while using fewer features or memory than some dense methods.
• Table 1 results show competitive feature counts and matches across methods, with D2 MS and variants achieving high match quality (D2 SS Trained and D2 MS Trained variants).
• The describe-and-detect approach yields robust correspondences under day-night and weakly textured scenarios, while maintaining reasonable localization and SfM performance despite less precise single-point localization compared to traditional detectors.