[论文解读] Learning Dense Correspondences between Photos and Sketches

Xuanchen Lu, Xiaolong Wang+1 more|arXiv (Cornell University)|Jul 24, 2023

Visual Attention and Saliency DetectionComputer Science被引用 3

一句话总结

本文提出了PSC6k，一个包含150K个人工标注的密集对应关系的新基准，覆盖6,250对草图-照片图像，共125个类别，并提出了一种基于对比学习和空间变换器的自监督方法，用于预测草图与照片之间的密集对应关系。该方法优于多个强基线模型，但揭示了模型预测与人类预测之间存在系统性差距，凸显了在跨模态场景下实现类人视觉理解的挑战。

ABSTRACT

Humans effortlessly grasp the connection between sketches and real-world objects, even when these sketches are far from realistic. Moreover, human sketch understanding goes beyond categorization -- critically, it also entails understanding how individual elements within a sketch correspond to parts of the physical world it represents. What are the computational ingredients needed to support this ability? Towards answering this question, we make two contributions: first, we introduce a new sketch-photo correspondence benchmark, $ extit{PSC6k}$, containing 150K annotations of 6250 sketch-photo pairs across 125 object categories, augmenting the existing Sketchy dataset with fine-grained correspondence metadata. Second, we propose a self-supervised method for learning dense correspondences between sketch-photo pairs, building upon recent advances in correspondence learning for pairs of photos. Our model uses a spatial transformer network to estimate the warp flow between latent representations of a sketch and photo extracted by a contrastive learning-based ConvNet backbone. We found that this approach outperformed several strong baselines and produced predictions that were quantitatively consistent with other warp-based methods. However, our benchmark also revealed systematic differences between predictions of the suite of models we tested and those of humans. Taken together, our work suggests a promising path towards developing artificial systems that achieve more human-like understanding of visual images at different levels of abstraction. Project page: https://photo-sketch-correspondence.github.io

研究动机与目标

• 为解决当前缺乏细粒度、人工标注的草图与照片之间对应关系数据以用于视觉理解的基准测试问题。
• 开发一种自监督方法，学习高度不同的图像模态（照片与草图）之间的密集语义对应关系，而无需依赖大规模人工标注的监督信号。
• 探究对比学习与空间变换器网络是否能有效建模跨模态对应关系，尽管存在显著的外观与抽象性差异。
• 将学习到的对应关系性能与人工标注的真实值进行评估，并识别出人类判断与模型预测之间的系统性差异。

提出的方法

• 该方法采用基于对比学习的卷积神经网络主干网络，从草图与照片配对中提取共享的潜在表征。
• 使用空间变换器网络预测一个密集的形变场，将照片特征图变换以对齐草图特征图。
• 模型通过自监督方式训练，结合对比损失与几何一致性目标，利用合成数据增强来模拟对应关系监督。
• 该框架利用预训练主干网络的层次化特征，并优化预测形变场的前向-后向一致性。
• 该方法端到端可微，支持特征提取与对应关系预测的联合优化。
• 该方法在新提出的PSC6k基准上进行评估，该基准为6,250对草图-照片图像提供了细粒度的关键点标注。

Figure 3: We propose a self-supervised framework for learning photo-sketch correspondence by estimating a dense displacement field that warps one image to the other. The framework consists of a multi-modal feature encoder that aligns the photo-sketch representation with a contrastive loss, and an ST

实验结果

研究问题

• RQ1 尽管存在较大的领域差距，自监督深度学习方法是否能有效学习照片与草图之间的密集语义对应关系？
• RQ2 深度学习模型的预测在定量和定性上与同一任务上的人工标注对应关系相比如何？
• RQ3 对比表示学习在多大程度上提升了照片与草图之间跨模态对应关系学习的零样本泛化能力？
• RQ4 当前模型在预测对应关系方面与人类判断相比，存在哪些系统性失败？

主要发现

• 所提出的自监督方法在PSC6k基准上预测草图与照片之间的密集对应关系时，优于多个强基线模型。
• 模型的预测在定量上与其他基于形变场的对应关系学习方法保持一致，表明其在形变估计方面具有鲁棒性。
• 尽管性能出色，模型在对齐细粒度部件以及处理草图中的抽象化方面仍表现出与人工标注对应关系的系统性偏差。
• PSC6k基准揭示，当前模型在处理草图中高度抽象化或被省略的部件时存在困难，表明其在建模语义抽象方面存在局限。
• 结果表明，尽管对比学习与空间变换器在跨模态对应关系学习中有效，但在复杂视觉抽象任务中仍无法完全复制人类水平的理解能力。

Figure 5: Measuring human and model consistency. Each cell represents the mean pixel distance between correspondence predictions generated by two systems (whether artificial or human), normalized by the image size. We denote models trained on Photo-sketch pairs with PS, and models trained on PF-Pasc