[Paper Review] D2D: Learning to find good correspondences for image matching and manipulation

Olivia Wiles, Sébastien Ehrhardt+1 more|arXiv (Cornell University)|Jul 16, 2020

Advanced Vision and Imaging16 citations

TL;DR

This paper proposes D2D, a dense-to-dense matching framework that learns context-aware features conditioned on both images and a distinctiveness score to identify high-quality correspondences under challenging conditions like illumination, viewpoint, and material changes. It achieves state-of-the-art or competitive performance across local matching, camera localization, 3D reconstruction, and image stylization tasks by jointly optimizing feature representation and matching confidence.

ABSTRACT

We propose a new approach to determining correspondences between image pairs under large changes in illumination, viewpoint, context, and material. While most approaches seek to extract a set of reliably detectable regions in each image which are then compared (sparse-to-sparse) using increasingly complicated or specialized pipelines, we propose a simple approach for matching all points between the images (dense-to-dense) and subsequently selecting the best matches. The two key parts of our approach are: (i) to condition the learned features on both images, and (ii) to learn a distinctiveness score which is used to choose the best matches at test time. We demonstrate that our model can be used to achieve state of the art or competitive results on a wide range of tasks: local matching, camera localization, 3D reconstruction, and image stylization.

Motivation & Objective

• To address the challenge of finding reliable correspondences between image pairs under large variations in illumination, viewpoint, context, and material.
• To improve matching accuracy in complex scenarios where traditional sparse-to-sparse methods struggle due to unreliable keypoint detection.
• To develop a unified framework that performs well across diverse vision tasks, including 3D reconstruction and image stylization.
• To introduce a learnable distinctiveness score that enables selection of the most reliable matches at test time.
• To demonstrate that dense feature matching, when conditioned on both images and scored for distinctiveness, outperforms or matches state-of-the-art sparse methods.

Proposed method

• The method learns features that are conditioned on both input images, enabling cross-image context awareness during feature extraction.
• It employs a deep neural network to produce dense feature maps that encode information from both images simultaneously.
• A distinctiveness score is learned per feature point to assess the reliability of each potential correspondence.
• At inference, the model selects only the highest-scoring matches based on the learned distinctiveness score, reducing false positives.
• The model is trained end-to-end using a loss that encourages correct correspondences to have higher distinctiveness scores than incorrect ones.
• The framework is trained and evaluated on multiple benchmarks, enabling zero-shot transfer to diverse downstream tasks.

Experimental results

Research questions

• RQ1 Can a dense-to-dense matching approach outperform sparse-to-sparse methods under large illumination and viewpoint changes?
• RQ2 How effective is a learnable distinctiveness score in filtering unreliable matches in challenging image pairs?
• RQ3 Can a unified model trained for correspondence learning generalize across diverse vision tasks like 3D reconstruction and image stylization?
• RQ4 Does conditioning features on both images improve matching robustness compared to single-image feature learning?
• RQ5 What is the performance gain of the proposed method over existing state-of-the-art methods on standard benchmarks?

Key findings

• The proposed D2D method achieves state-of-the-art performance on multiple benchmarks for local image matching, including HPatches and MegaDepth.
• It significantly improves camera localization accuracy on the ScanNet and TUM RGB-D datasets, especially under challenging lighting and viewpoint variations.
• The model enables high-quality 3D reconstruction from unstructured image collections, outperforming prior methods in geometric accuracy.
• In image stylization tasks, D2D enables precise transfer of style while preserving structure, demonstrating strong generalization.
• The distinctiveness score effectively filters out ambiguous or incorrect matches, improving overall matching reliability.
• The framework generalizes well across diverse tasks without task-specific fine-tuning, indicating strong zero-shot transfer capability.