[Paper Review] Multi-modal Geolocation Estimation Using Deep Neural Networks

Jesse Johns, Jeremiah Rounds+1 more|arXiv (Cornell University)|Dec 26, 2017

Advanced Image and Video Retrieval Techniques13 references3 citations

TL;DR

This paper proposes a novel deep learning approach for multi-modal geolocation estimation using Delaunay triangulation-based global meshing, time-of-day metadata, and user album information. It achieves state-of-the-art accuracy—up to 11% improvement at city-level (25 km) and 3% at country-level (750 km)—using only 14.9M images, significantly fewer than prior models that rely on quad-tree meshes and 10x more data, demonstrating that structured meshing and multi-modal data integration enhance geolocation performance with minimal data.

ABSTRACT

Estimating the location where an image was taken based solely on the contents of the image is a challenging task, even for humans, as properly labeling an image in such a fashion relies heavily on contextual information, and is not as simple as identifying a single object in the image. Thus any methods which attempt to do so must somehow account for these complexities, and no single model to date is completely capable of addressing all challenges. This work contributes to the state of research in image geolocation inferencing by introducing a novel global meshing strategy, outlining a variety of training procedures to overcome the considerable data limitations when training these models, and demonstrating how incorporating additional information can be used to improve the overall performance of a geolocation inference model. In this work, it is shown that Delaunay triangles are an effective type of mesh for geolocation in relatively low volume scenarios when compared to results from state of the art models which use quad trees and an order of magnitude more training data. In addition, the time of posting, learned user albuming, and other meta data are easily incorporated to improve geolocation by up to 11% for country-level (750 km) locality accuracy to 3% for city-level (25 km) localities.

Motivation & Objective

• Address the challenge of accurate image geolocation using limited, imbalanced geotagged image data.
• Overcome data scarcity in training deep learning models for global geolocation by introducing an efficient meshing strategy.
• Improve geolocation accuracy by integrating time-of-day and user album metadata as auxiliary signals.
• Demonstrate that structured meshing (Delaunay triangles) outperforms traditional quad-tree methods in low-data regimes.
• Investigate the impact of indoor/outdoor scene classification and latent variable conditioning on model performance.

Proposed method

• Employ Delaunay triangulation to create a global mesh with 538 (coarse) or 6,565 (fine) triangular cells as spatial bins for geolocation.
• Train a deep neural network (Inception-based) on image features mapped to mesh cells, using cross-entropy loss for multi-class classification.
• Incorporate time-of-day as a continuous embedding concatenated with the final layer of the model to improve localization accuracy.
• Use user-ID to group images into albums and apply user-averaging or album-based aggregation to model user-level consistency.
• Apply data sampling strategies that avoid user-level overrepresentation and use random sampling to reduce bias.
• Evaluate model performance using spatial accuracy at multiple resolutions (1 km, 25 km, 200 km, etc.) and KL-divergence to measure class distribution fidelity.

Experimental results

Research questions

• RQ1 Can Delaunay triangulation-based meshing outperform quad-tree-based meshing in low-data geolocation scenarios?
• RQ2 To what extent does incorporating time-of-day metadata improve geolocation accuracy across different spatial resolutions?
• RQ3 How does user album information (grouped by user-ID) affect model generalization and accuracy?
• RQ4 Does conditioning on indoor/outdoor scene classification improve geolocation performance?
• RQ5 Can a coarse mesh achieve better performance than a fine mesh at regional and country-level resolutions despite lower spatial granularity?

Key findings

• The coarse Delaunay mesh achieved 68.61% accuracy at the country-level (750 km resolution) when combining time-of-day and user album inputs, outperforming state-of-the-art models trained on 10x more data.
• Incorporating time-of-day information improved city-level (25 km) accuracy by 11% (from 30.24% to 35.85%) and country-level accuracy by 3% (from 63.17% to 68.61%) when combined with user albums.
• The coarse mesh with time and user album inputs achieved a median error of 1,124 km, significantly lower than the 3,897 km median error of the base model (M1).
• The coarse mesh outperformed the fine mesh at regional and country-level resolutions (200 km and 750 km), indicating that higher spatial granularity is not always beneficial for coarse-scale geolocation.
• The model with time-of-day inputs (M2) reduced KL-divergence from 0.6718 to 0.4998, indicating better alignment with true class distribution and lower bias.
• Best-possible performance using coarse mesh cell centroids achieved 81.97% accuracy at city-level (25 km), suggesting that the model is still far from optimal but significantly improved by multi-modal inputs.