[Paper Review] 3rd Place Solution to Google Landmark Recognition Competition 2021

Cheng Xu, Weimin Wang+7 more|arXiv (Cornell University)|Oct 6, 2021

Domain Adaptation and Few-Shot Learning13 references4 citations

TL;DR

This paper presents a 3rd-place solution to the 2021 Google Landmark Recognition Competition, leveraging multi-architecture models (CNN, Transformer, and hybrid) with ArcFace loss for feature embedding. The method employs a re-ranking pipeline combining retrieval scores, classification logits, distractor score penalization, and top-1 classification aggregation, achieving 0.48962 on the private leaderboard.

ABSTRACT

In this paper, we show our solution to the Google Landmark Recognition 2021 Competition. Firstly, embeddings of images are extracted via various architectures (i.e. CNN-, Transformer- and hybrid-based), which are optimized by ArcFace loss. Then we apply an efficient pipeline to re-rank predictions by adjusting the retrieval score with classification logits and non-landmark distractors. Finally, the ensembled model scores 0.489 on the private leaderboard, achieving the 3rd place in the 2021 edition of the Google Landmark Recognition Competition.

Motivation & Objective

• To address instance-level landmark recognition in a large-scale, imbalanced dataset with high intra-class variation and non-landmark images.
• To improve retrieval accuracy by combining diverse model architectures (CNN, Transformer, hybrid) for complementary feature learning.
• To enhance prediction reliability by integrating classification logits and penalizing predictions based on non-landmark similarity.
• To optimize model ensembling through normalized concatenation of embeddings and end-to-end re-ranking.
• To achieve state-of-the-art performance on the Google Landmark Recognition 2021 competition using a robust, scalable pipeline.

Proposed method

• Extracted 512D image embeddings using multiple backbones: EfficientNet-B5/B6/B7/V2 (CNN), Swin-L-384 (Transformer), and CvT-W24-384 (hybrid).
• Optimized embeddings via ArcFace loss with adaptive margin, using GeM pooling for CNNs and direct token pooling for Transformers.
• Applied a multi-stage training schedule: first on GLDv2c (1.5M images, 81K classes), then GLDv2x (3.2M), and finally GLDv2 (4.1M, 203K classes) with frozen backbones and fine-tuned heads.
• Used data augmentation (RandAug, CutOut, RandomResizedCrop) with resolution scaling and learning rate decay via cosine annealing.
• Implemented a re-ranking pipeline: adjusted retrieval scores by adding classification logits and subtracting distractor scores from non-landmark matches.
• Ensembled models by concatenating L2-normalized embeddings and re-running the full inference pipeline on the combined space.

Experimental results

Research questions

• RQ1 How can diverse model architectures (CNN, Transformer, hybrid) be effectively combined to improve landmark retrieval performance?
• RQ2 To what extent do classification logits enhance retrieval accuracy, especially for rare landmarks with few training examples?
• RQ3 Can non-landmark image similarity scores be used as an effective penalization mechanism to reduce false positives?
• RQ4 How does top-1 classification logit aggregation improve final predictions when combined with retrieval and distractor adjustments?
• RQ5 What training schedule and data partitioning strategy maximizes generalization and generalization to unseen landmarks?

Key findings

• The Swin-L-384 model achieved the highest retrieval score of 0.383 on the public leaderboard, outperforming all CNN-based models.
• Ensembling Swin-L and CvT-W24-384 led to a more significant performance gain than ensembling CNN models alone, indicating strong architectural complementarity.
• The final model achieved a private leaderboard score of 0.48962, ranking 3rd in the competition.
• Classification logit adjustment improved predictions, particularly for low-frequency landmarks, with B5 achieving ~0.39 top-1 accuracy on public LB.
• Distractor score penalization effectively reduced false positives on non-landmark images, improving robustness in retrieval.
• Top-1 classification aggregation provided an additional boost, even without distractor penalization, due to its natural ability to suppress non-landmark predictions.