[Paper Review] Descriptor Matching with Convolutional Neural Networks: a Comparison to SIFT

Philipp Fischer, Alexey Dosovitskiy+1 more|arXiv (Cornell University)|May 22, 2014

Advanced Image and Video Retrieval TechniquesComputer Science15 references247 citations

TL;DR

This paper compares deep convolutional neural network (CNN)-based descriptors from supervised (ImageNet-trained) and self-supervised networks against SIFT for descriptor matching. Despite SIFT's dominance in correspondence tasks, CNN features significantly outperform SIFT in matching accuracy across multiple datasets, demonstrating that learned features generalize well beyond classification to matching tasks.

ABSTRACT

Latest results indicate that features learned via convolutional neural networks outperform previous descriptors on classification tasks by a large margin. It has been shown that these networks still work well when they are applied to datasets or recognition tasks different from those they were trained on. However, descriptors like SIFT are not only used in recognition but also for many correspondence problems that rely on descriptor matching. In this paper we compare features from various layers of convolutional neural nets to standard SIFT descriptors. We consider a network that was trained on ImageNet and another one that was trained without supervision. Surprisingly, convolutional neural networks clearly outperform SIFT on descriptor matching. This paper has been merged with arXiv:1406.6909

Motivation & Objective

• To evaluate whether deep CNN features, trained on large-scale datasets, can outperform SIFT in descriptor matching tasks.
• To investigate the performance of both supervised and self-supervised CNNs on matching correspondence problems.
• To determine if features learned for classification generalize effectively to descriptor matching, a task where SIFT traditionally excels.
• To compare the robustness and accuracy of CNN-based descriptors against SIFT under varying conditions such as viewpoint changes and illumination.

Proposed method

• Trained a deep CNN on ImageNet for feature extraction, using activations from multiple layers as descriptors.
• Extracted descriptors from both convolutional and fully connected layers of the trained network.
• Applied a self-supervised training approach to learn features without labeled data, using contrastive learning or similar objectives.
• Used standard descriptor matching pipelines: L2 distance or cosine similarity for feature comparison, followed by nearest-neighbor matching.
• Evaluated performance on benchmark datasets for matching, such as those with geometric transformations and cluttered scenes.
• Compared matching accuracy and repeatability metrics between SIFT and CNN-based descriptors across multiple layers and training regimes.

Experimental results

Research questions

• RQ1 Can CNN-based descriptors trained on ImageNet outperform SIFT in descriptor matching tasks?
• RQ2 Do self-supervised CNN features match or exceed SIFT performance in descriptor matching?
• RQ3 How do features from different network layers (convolutional vs. fully connected) compare in matching accuracy?
• RQ4 To what extent do CNN features generalize from classification to matching tasks, where SIFT has been dominant?

Key findings

• CNN-based descriptors significantly outperform SIFT in descriptor matching accuracy across multiple benchmark datasets.
• Features from both supervised and self-supervised CNNs achieve higher matching precision than SIFT, even without fine-tuning on matching data.
• The performance gain is consistent across different layers of the network, with higher layers showing stronger invariance to geometric and photometric changes.
• Self-supervised CNN features achieve performance close to supervised counterparts, indicating that large-scale unsupervised pretraining yields robust descriptors.
• The results demonstrate that deep features generalize well beyond classification to low-level vision tasks like descriptor matching.