[Paper Review] Learning Dense Correspondences between Photos and Sketches

Xuanchen Lu, Xiaolong Wang+1 more|arXiv (Cornell University)|Jul 24, 2023

Visual Attention and Saliency DetectionComputer Science3 citations

TL;DR

This paper introduces PSC6k, a new benchmark with 150K human-annotated dense correspondences across 6,250 sketch-photo pairs in 125 categories, and proposes a self-supervised method using contrastive learning and a spatial transformer to predict dense correspondences between sketches and photos. The approach outperforms strong baselines but reveals systematic gaps between model and human predictions, highlighting challenges in achieving human-like visual understanding across modalities.

ABSTRACT

Humans effortlessly grasp the connection between sketches and real-world objects, even when these sketches are far from realistic. Moreover, human sketch understanding goes beyond categorization -- critically, it also entails understanding how individual elements within a sketch correspond to parts of the physical world it represents. What are the computational ingredients needed to support this ability? Towards answering this question, we make two contributions: first, we introduce a new sketch-photo correspondence benchmark, $ extit{PSC6k}$, containing 150K annotations of 6250 sketch-photo pairs across 125 object categories, augmenting the existing Sketchy dataset with fine-grained correspondence metadata. Second, we propose a self-supervised method for learning dense correspondences between sketch-photo pairs, building upon recent advances in correspondence learning for pairs of photos. Our model uses a spatial transformer network to estimate the warp flow between latent representations of a sketch and photo extracted by a contrastive learning-based ConvNet backbone. We found that this approach outperformed several strong baselines and produced predictions that were quantitatively consistent with other warp-based methods. However, our benchmark also revealed systematic differences between predictions of the suite of models we tested and those of humans. Taken together, our work suggests a promising path towards developing artificial systems that achieve more human-like understanding of visual images at different levels of abstraction. Project page: https://photo-sketch-correspondence.github.io

Motivation & Objective

• To address the lack of fine-grained, human-annotated correspondence data between sketches and photos for benchmarking visual understanding.
• To develop a self-supervised method that learns dense semantic correspondences between highly dissimilar image modalities—photos and sketches—without relying on large-scale human-annotated supervision.
• To investigate whether contrastive learning and spatial transformer networks can effectively model cross-modal correspondence despite significant domain shift in appearance and abstraction.
• To evaluate the performance of learned correspondences against human-annotated ground truth and identify systematic discrepancies between human and model predictions.

Proposed method

• The method employs a contrastive learning-based ConvNet backbone to extract shared latent representations from sketch and photo pairs.
• A spatial transformer network is used to predict a dense warp flow field that transforms the photo feature map to align with the sketch feature map.
• The model is trained in a self-supervised manner using a combination of contrastive loss and geometric consistency objectives, leveraging synthetic data augmentations to simulate correspondence supervision.
• The framework leverages hierarchical features from a pre-trained backbone and optimizes for forward-backward consistency in the predicted flow fields.
• The approach is end-to-end differentiable, enabling joint optimization of feature extraction and correspondence prediction.
• The method is evaluated on the newly introduced PSC6k benchmark, which provides fine-grained keypoint annotations for 6,250 sketch-photo pairs.

Figure 3: We propose a self-supervised framework for learning photo-sketch correspondence by estimating a dense displacement field that warps one image to the other. The framework consists of a multi-modal feature encoder that aligns the photo-sketch representation with a contrastive loss, and an ST

Experimental results

Research questions

• RQ1 Can a self-supervised deep learning method effectively learn dense semantic correspondences between photos and sketches, despite their large domain gap?
• RQ2 How do the predictions of deep learning models compare quantitatively and qualitatively to human-annotated correspondences on the same task?
• RQ3 To what extent does contrastive representation learning improve zero-shot generalization for cross-modal correspondence learning between photos and sketches?
• RQ4 What are the systematic failures of current models in predicting correspondences compared to human judgments?

Key findings

• The proposed self-supervised method outperformed several strong baselines in predicting dense correspondences between sketches and photos on the PSC6k benchmark.
• The model's predictions were quantitatively consistent with other warp-based correspondence learning methods, indicating robustness in flow estimation.
• Despite strong performance, the model exhibited systematic deviations from human-annotated correspondences, particularly in aligning fine-grained parts and handling abstraction in sketches.
• The PSC6k benchmark revealed that current models struggle with parts that are highly abstracted or omitted in sketches, suggesting limitations in modeling semantic abstraction.
• The results indicate that while contrastive learning and spatial transformers are effective for cross-modal correspondence, they fall short of replicating human-level understanding in complex visual abstraction tasks.

Figure 5: Measuring human and model consistency. Each cell represents the mean pixel distance between correspondence predictions generated by two systems (whether artificial or human), normalized by the image size. We denote models trained on Photo-sketch pairs with PS, and models trained on PF-Pasc