[Paper Review] Learning to Compare Image Patches via Convolutional Neural Networks

Sergey Zagoruyko, Nikos Komodakis|arXiv (Cornell University)|Apr 14, 2015

Advanced Image and Video Retrieval Techniques23 references197 citations

TL;DR

This paper proposes a convolutional neural network (CNN)-based approach to learn a general similarity function for comparing image patches directly from raw pixel data, without relying on handcrafted features like SIFT. The method uses siamese and 2-channel CNN architectures trained on large datasets of patch pairs, achieving state-of-the-art performance on benchmark tasks including wide baseline stereo, feature matching, and image retrieval, with significant improvements over SIFT and other learned descriptors.

ABSTRACT

In this paper we show how to learn directly from image data (i.e., without resorting to manually-designed features) a general similarity function for comparing image patches, which is a task of fundamental importance for many computer vision problems. To encode such a function, we opt for a CNN-based model that is trained to account for a wide variety of changes in image appearance. To that end, we explore and study multiple neural network architectures, which are specifically adapted to this task. We show that such an approach can significantly outperform the state-of-the-art on several problems and benchmark datasets.

Motivation & Objective

• To develop a general-purpose similarity function for image patches that learns directly from raw image data without hand-designed features.
• To explore and optimize deep neural network architectures specifically tailored for patch comparison under diverse appearance variations.
• To outperform existing handcrafted (e.g., SIFT) and learned descriptors on standard benchmarks for feature matching and image retrieval.
• To enable efficient, dense descriptor computation via the convolutional nature of the proposed models.

Proposed method

• The model uses a siamese or 2-channel CNN architecture to compare two input image patches by processing them through shared or separate branches.
• The network is trained on a large dataset of paired image patches (matching and non-matching) using a contrastive loss or similar objective to learn discriminative features.
• A 2-channel network processes both patches simultaneously and outputs a similarity score, enabling efficient inference.
• SPP (Spatial Pyramid Pooling) layers are incorporated into siamese models to aggregate multi-scale features, improving robustness to scale and deformation.
• Multi-resolution 2-stream architectures are used to capture features at different scales, enhancing performance on challenging matching tasks.
• The network is trained end-to-end using backpropagation, with no reliance on pre-trained features or manual feature engineering.

Experimental results

Research questions

• RQ1 Can a deep CNN learn a general-purpose similarity function for image patches directly from raw pixels, without handcrafted features?
• RQ2 Which neural network architecture—siamese, 2-channel, or SPP-enhanced—yields the best performance for patch comparison under appearance variations?
• RQ3 How does the proposed method compare to SIFT and other learned descriptors on standard benchmarks like Mikolajczyk and KITTI?
• RQ4 To what extent does multi-scale feature extraction improve patch matching robustness?
• RQ5 Can performance be further improved by enlarging the training dataset of patch pairs?

Key findings

• The 2-channel CNN architecture consistently outperformed all other models, including siamese and SPP-based variants, on both stereo matching and descriptor evaluation tasks.
• The siamese-2stream-l2 model achieved performance comparable to ImageNet-pretrained features but with a much lower descriptor dimensionality of 512.
• SPP-based siamese networks showed a significant performance boost, indicating the value of multi-scale feature aggregation for patch comparison.
• The proposed method outperformed SIFT and DAISY on the Mikolajczyk dataset, achieving higher mean average precision (mAP) across all transformation types.
• On the KITTI stereo dataset, the siamese models significantly reduced error rates compared to DAISY, especially at higher disparity thresholds (3 and 5 pixels).
• The results suggest that increasing the size of the training dataset could further improve performance, as the current dataset is considered relatively small by modern standards.