[Paper Review] TransGeo: Transformer Is All You Need for Cross-view Image Geo-localization

Sijie Zhu, Mubarak Shah+1 more|arXiv (Cornell University)|Mar 31, 2022

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

TransGeo proposes a pure transformer-based method for cross-view image geo-localization that eliminates reliance on polar transforms and data augmentation. By using attention-guided non-uniform cropping to focus computation on informative image patches and reallocating resources to higher resolution in those regions, it achieves state-of-the-art performance with significantly reduced FLOPs and inference time compared to CNN-based methods.

ABSTRACT

The dominant CNN-based methods for cross-view image geo-localization rely on polar transform and fail to model global correlation. We propose a pure transformer-based approach (TransGeo) to address these limitations from a different perspective. TransGeo takes full advantage of the strengths of transformer related to global information modeling and explicit position information encoding. We further leverage the flexibility of transformer input and propose an attention-guided non-uniform cropping method, so that uninformative image patches are removed with negligible drop on performance to reduce computation cost. The saved computation can be reallocated to increase resolution only for informative patches, resulting in performance improvement with no additional computation cost. This "attend and zoom-in" strategy is highly similar to human behavior when observing images. Remarkably, TransGeo achieves state-of-the-art results on both urban and rural datasets, with significantly less computation cost than CNN-based methods. It does not rely on polar transform and infers faster than CNN-based methods. Code is available at https://github.com/Jeff-Zilence/TransGeo2022.

Motivation & Objective

• To address the limitations of CNN-based cross-view geo-localization methods that rely on polar transforms and struggle with global correlation modeling.
• To leverage the global modeling and explicit position encoding capabilities of vision transformers for improved geo-localization.
• To develop a flexible, computation-efficient method that avoids uninformative image patches without sacrificing performance.
• To enable faster inference and reduced memory usage through adaptive, non-uniform cropping based on attention maps.
• To demonstrate that pure transformers can outperform CNNs in cross-view geo-localization without domain-specific preprocessing or data augmentation.

Proposed method

• Proposes a pure vision transformer architecture (TransGeo) for cross-view image geo-localization, replacing CNNs and eliminating the need for polar transformation.
• Introduces learnable 2D position embeddings to explicitly encode spatial relationships between image patches, improving geometric correspondence learning.
• Employs attention-guided non-uniform cropping: the model first computes attention maps from the last transformer encoder layer to identify informative patches, then increases resolution only on those regions.
• Reallocates saved computation from uninformative patches to higher-resolution processing on key regions, enabling performance gains without additional FLOPs.
• Uses Adaptive Sharpness-Aware Minimization (ASAM) to improve generalization and reduce overfitting, especially on medium-scale datasets.
• Applies a two-stage training pipeline: first train with full input, then fine-tune with cropped and zoomed-in patches guided by attention maps.

Experimental results

Research questions

• RQ1 Can a pure transformer-based method outperform CNN-based approaches in cross-view image geo-localization without relying on polar transforms?
• RQ2 Can attention maps be effectively used to guide non-uniform cropping to reduce computation while preserving or improving performance?
• RQ3 Does the use of learnable position embeddings in vision transformers improve performance on cross-view geo-localization with large domain gaps?
• RQ4 Can computation savings from removing uninformative patches be reallocated to increase resolution in informative regions, leading to performance gains with no additional FLOP cost?
• RQ5 How does the proposed 'attend and zoom-in' strategy compare to human visual attention in image understanding?

Key findings

• TransGeo achieves state-of-the-art performance on both urban (CVUSA) and rural (VIGOR) datasets, outperforming prior CNN-based methods like SAFA.
• On CVUSA, TransGeo achieves a 94.08% recall@1 with 320x320 resolution and 0.64 patch retention rate, outperforming the baseline by 1.0% with reduced FLOPs.
• The attention-guided non-uniform cropping removes 36% of patches (β=0.64) with only a 0.36% drop in R@1 on VIGOR and 0.1% on CVUSA, proving uninformative patches can be safely discarded.
• Reallocating computation to increase resolution (γ=1.56) on selected patches improves R@1 by 1.0% on CVUSA compared to the baseline, with no increase in total FLOPs.
• Learnable position embeddings significantly outperform fixed sinusoidal embeddings, improving R@1 from 42.72% to 93.18% on CVUSA, demonstrating their importance in cross-view alignment.
• Visualization shows strong global correlation learning in attention maps, especially in deeper layers, confirming the transformer’s ability to model long-range dependencies across views.