[Paper Review] Emergent Correspondence from Image Diffusion

Luming Tang, Menglin Jia+3 more|arXiv (Cornell University)|Jun 6, 2023

Advanced Neuroimaging Techniques and Applications54 citations

TL;DR

The paper shows that diffusion models implicitly learn pixel correspondences and introduces Diffusion Features (DIFT) to establish semantic, geometric, and temporal correspondences on real images without supervision.

ABSTRACT

Finding correspondences between images is a fundamental problem in computer vision. In this paper, we show that correspondence emerges in image diffusion models without any explicit supervision. We propose a simple strategy to extract this implicit knowledge out of diffusion networks as image features, namely DIffusion FeaTures (DIFT), and use them to establish correspondences between real images. Without any additional fine-tuning or supervision on the task-specific data or annotations, DIFT is able to outperform both weakly-supervised methods and competitive off-the-shelf features in identifying semantic, geometric, and temporal correspondences. Particularly for semantic correspondence, DIFT from Stable Diffusion is able to outperform DINO and OpenCLIP by 19 and 14 accuracy points respectively on the challenging SPair-71k benchmark. It even outperforms the state-of-the-art supervised methods on 9 out of 18 categories while remaining on par for the overall performance. Project page: https://diffusionfeatures.github.io

Motivation & Objective

• Demonstrate that image diffusion models learn correspondences without explicit supervision.
• Extract dense diffusion-based features (DIFT) from pre-trained diffusion models for real images.
• Evaluate DIFT on semantic, geometric, and temporal correspondence benchmarks and compare with self-supervised and supervised methods.
• Show practical applications such as edit propagation and cross-domain matching.

Proposed method

• Use pre-trained diffusion models (Stable Diffusion 2-1 and Ablated Diffusion Model) to extract intermediate-layer activations during the backward diffusion process.
• Approximate real-image features by adding noise to real images to reach a diffusion timestep t and passing them through the diffusion model to obtain DIFT.
• Select time step t and network layer via 2D grid search per task to balance semantic vs. low-level information.
• Average features over multiple noisy realizations to improve stability.
• Perform nearest-neighbor matching with cosine distance between pixel features to establish correspondences.
• Evaluate across semantic, geometric, and temporal correspondence benchmarks without any fine-tuning or supervision.

Figure 1 : Given a red source point in an image (far left), we would like to develop a model that automatically finds the corresponding point in the images on the right. Without any fine-tuning or correspondence supervision, our proposed diffusion features (DIFT) could establish semantic corresponde

Experimental results

Research questions

• RQ1 Can diffusion models trained for image synthesis implicitly acquire pixel-level correspondences useful for cross-image matching?
• RQ2 How can we extract dense, real-image features from diffusion models without fine-tuning for correspondence tasks?
• RQ3 What are the optimal diffusion timestep t and layer for different correspondence tasks (semantic, geometric, temporal)?
• RQ4 How does DIFT compare to self-supervised and supervised methods on standard correspondence benchmarks?
• RQ5 Can DIFT enable tasks like edit propagation and video/object tracking without supervision?

Key findings

• DIFT enables robust semantic, geometric, and temporal correspondences on real images without any task-specific supervision.
• On SPair-71k, DIFT sd outperforms DINO and OpenCLIP in semantic PCK by 19 and 14 points respectively; it is competitive with supervised methods on semantic correspondence across 9 of 18 categories.
• DIFT outperforms weakly-supervised baselines and self-supervised features across benchmarks PF-WILLOW and SPair-71k, with semantic results sometimes surpassing state-of-the-art supervised methods.
• DIFT provides strong geometric correspondence performance on HPatches, comparable to geometry-supervised methods when using SuperPoint keypoints.
• In temporal tasks, DIFT achieves state-of-the-art or strong results on DAVIS-2017 and JHMDB without video-specific training, indicating rich cross-domain generalization.
• Edit propagation experiments show DIFT-based matching yields more accurate cross-image edits than OpenCLIP.

Figure 2 : Given a Stable Diffusion generated image, we extract its intermediate layer activations at a certain time step $t$ during its backward process, and use them as the feature map to predict the corresponding points. Although simple, this method produces correct correspondences on generated i