[论文解读] Cross-view Geo-localization with Evolving Transformer

Hongji Yang, Xiufan Lu+1 more|arXiv (Cornell University)|Jul 2, 2021

Advanced Neural Network Applications参考文献 25被引用 6

一句话总结

该论文提出EgoTR，一种基于Transformer的新型模型，用于跨视角地理定位。该模型利用自交叉注意力机制增强网络深层之间的特征演化，并采用可学习的位置嵌入来建模几何配置。在标准、细粒度和跨数据集基准上，EgoTR在无需强几何假设的前提下，优于当前最先进方法。

ABSTRACT

In this work, we address the problem of cross-view geo-localization, which estimates the geospatial location of a street view image by matching it with a database of geo-tagged aerial images. The cross-view matching task is extremely challenging due to drastic appearance and geometry differences across views. Unlike existing methods that predominantly fall back on CNN, here we devise a novel evolving geo-localization Transformer (EgoTR) that utilizes the properties of self-attention in Transformer to model global dependencies, thus significantly decreasing visual ambiguities in cross-view geo-localization. We also exploit the positional encoding of Transformer to help the EgoTR understand and correspond geometric configurations between ground and aerial images. Compared to state-of-the-art methods that impose strong assumption on geometry knowledge, the EgoTR flexibly learns the positional embeddings through the training objective and hence becomes more practical in many real-world scenarios. Although Transformer is well suited to our task, its vanilla self-attention mechanism independently interacts within image patches in each layer, which overlooks correlations between layers. Instead, this paper propose a simple yet effective self-cross attention mechanism to improve the quality of learned representations. The self-cross attention models global dependencies between adjacent layers, which relates between image patches while modeling how features evolve in the previous layer. As a result, the proposed self-cross attention leads to more stable training, improves the generalization ability and encourages representations to keep evolving as the network goes deeper. Extensive experiments demonstrate that our EgoTR performs favorably against state-of-the-art methods on standard, fine-grained and cross-dataset cross-view geo-localization tasks.

研究动机与目标

• 为解决跨视角地理定位中的挑战，即地面视图与航空视图之间存在显著的外观和几何差异，导致视觉模糊。
• 克服基于CNN的模型依赖局部归纳偏置、在视觉干扰和复杂场景中表现不佳的局限性。
• 消除先前方法中使用的强几何假设（例如方向对齐），以提升在更广泛真实场景中的适用性。
• 通过一种新颖的自交叉注意力机制建模跨层特征演化，提升深度Transformer中的表征学习能力。

提出的方法

• EgoTR采用两个独立的视觉Transformer分支分别处理地面图像和航空图像，利用自注意力机制建模全局上下文。
• 引入可学习的位置编码，使模型能够在无需先验几何假设的前提下理解并对应不同视图之间的几何布局。
• 提出一种新颖的自交叉注意力机制，通过计算相邻Transformer块之间特征的成对相似性，替代标准自注意力机制。
• 该机制增强了层间信息流动，减少了表征冗余，并促进了特征表示随深度逐步演化。
• 模型通过对比损失端到端训练，以优化地面图像与航空图像对之间的匹配性能。
• 消融实验验证了自交叉注意力和可学习位置嵌入在性能与泛化能力上的贡献。

实验结果

研究问题

• RQ1 基于Transformer的架构是否能通过建模长距离依赖和全局上下文，在跨视角地理定位中超越基于CNN的模型？
• RQ2 可学习的位置嵌入是否能有效捕捉跨视角的几何配置，而无需施加强几何先验？
• RQ3 与每个块中独立的自注意力相比，跨层注意力交互是否能提升表征质量和训练稳定性？
• RQ4 自交叉注意力机制如何影响特征演化与泛化能力，特别是在少样本学习场景下？

主要发现

• 在CVUSA数据集上，EgoTR实现了94.05%的top-1准确率，较使用标准自注意力的基线模型提升了0.79%。
• 在少样本跨视角地理定位任务中，无论训练数据占比为20%、40%还是60%，用自交叉注意力替代自注意力均能持续提升性能。
• 余弦相似性分析表明，自交叉注意力显著降低了层间表征的相似性，表明特征表示更加独特且持续演化。
• 定性分析表明，学习到的位置嵌入能够捕捉相对空间偏移，并反映透视失真现象，如图像中心附近物体间距的变化。
• 模型的位置编码成功编码了跨视角的几何对齐关系，证据显示在极坐标变换后，EgoTR的位置嵌入呈现出明显不同的模式，而Polar-EgoTR则表现出相似的模式。