[Paper Review] Towards A Fairer Landmark Recognition Dataset

Zu Kim, André Araujo+6 more|arXiv (Cornell University)|Aug 19, 2021

Advanced Image and Video Retrieval Techniques6 references4 citations

TL;DR

This paper proposes a fairer landmark recognition dataset by estimating landmark relevance using stratified sampling of anonymized Google Maps user contributions, adjusted with global demographic statistics to reduce geographic and socioeconomic bias. The resulting dataset achieves significantly improved representation of under-represented regions—particularly Africa, India, and the Middle East—compared to prior datasets like GLDv2, as validated through benchmark evaluations in the 2021 Google Landmark Challenges.

ABSTRACT

We introduce a new landmark recognition dataset, which is created with a focus on fair worldwide representation. While previous work proposes to collect as many images as possible from web repositories, we instead argue that such approaches can lead to biased data. To create a more comprehensive and equitable dataset, we start by defining the fair relevance of a landmark to the world population. These relevances are estimated by combining anonymized Google Maps user contribution statistics with the contributors' demographic information. We present a stratification approach and analysis which leads to a much fairer coverage of the world, compared to existing datasets. The resulting datasets are used to evaluate computer vision models as part of the the Google Landmark Recognition and RetrievalChallenges 2021.

Motivation & Objective

• Address the geographic and socioeconomic bias present in existing landmark recognition datasets collected from web repositories.
• Improve representation of under-represented regions such as Africa, India, and the Middle East in landmark datasets.
• Develop a method to estimate fair landmark relevance by combining user contribution statistics with world population demographics.
• Create evaluation datasets for the 2021 Google Landmark Recognition and Retrieval Challenges that reflect more equitable global coverage.
• Provide a baseline evaluation to serve as a reference for future model development in fair landmark recognition.

Proposed method

• Define fair landmark relevance based on two dimensions: personal utility and tourism appeal, to reflect global population relevance.
• Use anonymized and aggregated Google Maps user contribution data (e.g., image uploads, ratings) as a proxy for landmark popularity.
• Apply stratified sampling to adjust for demographic imbalances in contributors by modeling $ P(c) = \sum_h P(c|h)P(h) $, where $ h $ represents demographic groups.
• Estimate $ P(h) $ using world population statistics (e.g., gender, country, age) to correct for over- or under-representation of certain groups.
• Compute relevance scores for 50,000 landmarks using $ P(c|h) = A_{c,h}/A_h $, where $ A_{c,h} $ is contributions to landmark $ c $ by group $ h $.
• Resample images from GLDv2 and Wikimedia based on relevance scores to construct new evaluation datasets with fairer global distribution.

Experimental results

Research questions

• RQ1 How can landmark relevance be estimated in a way that reflects global population distribution rather than biased contributor demographics?
• RQ2 To what extent does stratified sampling of user contributions reduce geographic bias in landmark recognition datasets?
• RQ3 Can a fairer evaluation dataset be constructed for landmark recognition and retrieval challenges using demographic-adjusted relevance scores?
• RQ4 How does model performance on the new fair dataset compare to previous benchmarks in terms of fairness and generalization?
• RQ5 What are the limitations of using web-based contribution data for estimating global landmark relevance despite demographic adjustments?

Key findings

• The proposed stratified approach significantly improves representation of under-represented continents—especially Africa and South Asia—compared to GLDv2, as shown in the continent distribution of top-5000 landmarks.
• The top-10 landmarks in the stratified dataset include globally significant sites from diverse regions such as Masjid al-Haram (Saudi Arabia), the Taj Mahal (India), and the Giza Necropolis (Egypt), reducing over-representation of Western landmarks.
• The new evaluation datasets for the 2021 Google Landmark Challenges were constructed by resampling images based on demographic-adjusted relevance scores, ensuring fairer coverage of landmarks from all regions.
• Baseline results using DELG embeddings show a slight improvement in mAP@100 for both recognition (0.2303 private, 0.2376 public) and retrieval (0.2223 private, 0.2153 public) tasks compared to 2020 benchmarks.
• Despite improvements, the method still faces limitations due to insufficient data from regions with low internet access and incomplete relevance scores for some landmarks, especially those not present in both Google Maps and Wikimedia.