[Paper Review] Improving Image Classification with Location Context

Kevin Tang, Manohar Paluri+3 more|arXiv (Cornell University)|May 14, 2015

Advanced Image and Video Retrieval Techniques38 references4 citations

TL;DR

This paper proposes integrating GPS-derived location context into convolutional neural networks (CNNs) for improved image classification. By designing five novel location features and enabling end-to-end learning of optimal pooling radii within the CNN, the authors achieve a 7% relative gain in mean average precision on a new geotagged dataset, demonstrating that location context significantly enhances recognition of location-sensitive visual concepts.

ABSTRACT

With the widespread availability of cellphones and cameras that have GPS capabilities, it is common for images being uploaded to the Internet today to have GPS coordinates associated with them. In addition to research that tries to predict GPS coordinates from visual features, this also opens up the door to problems that are conditioned on the availability of GPS coordinates. In this work, we tackle the problem of performing image classification with location context, in which we are given the GPS coordinates for images in both the train and test phases. We explore different ways of encoding and extracting features from the GPS coordinates, and show how to naturally incorporate these features into a Convolutional Neural Network (CNN), the current state-of-the-art for most image classification and recognition problems. We also show how it is possible to simultaneously learn the optimal pooling radii for a subset of our features within the CNN framework. To evaluate our model and to help promote research in this area, we identify a set of location-sensitive concepts and annotate a subset of the Yahoo Flickr Creative Commons 100M dataset that has GPS coordinates with these concepts, which we make publicly available. By leveraging location context, we are able to achieve almost a 7% gain in mean average precision.

Motivation & Objective

• To improve image classification performance by incorporating GPS coordinates as contextual priors.
• To design effective, data-driven features from GPS coordinates that capture geographic context beyond latitude and longitude.
• To develop a CNN architecture that jointly learns visual and location-based features with end-to-end optimization.
• To enable automatic learning of optimal spatial pooling radii for location features within the deep learning framework.
• To create and release a publicly available, large-scale, geotagged image dataset (YFCC100M-GEO100) for evaluating location-aware image classification.

Proposed method

• Propose five types of location context features: hashtag context, visual context, land cover, elevation, and climate, derived from GPS coordinates and external geographic datasets.
• Integrate these features into a CNN by concatenating them with visual features at the fully connected layer, followed by a post-connection layer to model cross-feature interactions.
• Replace fixed-radius histogram pooling with multiple replicas of a learnable radius layer (RL10) to automatically determine optimal spatial scales for feature aggregation.
• Train the entire network end-to-end using stochastic gradient descent, allowing joint optimization of visual, location, and radius parameters.
• Use a 256-dimensional fully connected layer as the final representation before classification, avoiding overfitting seen with deeper or larger layers.
• Leverage the YFCC100M-GEO100 dataset—88,986 geotagged images across 100 location-sensitive concepts—for training and evaluation.

Experimental results

Research questions

• RQ1 Can GPS coordinates significantly improve image classification accuracy when used as contextual priors?
• RQ2 Which types of location-derived features (e.g., land cover, elevation, climate) are most effective for improving visual recognition?
• RQ3 Can a deep learning model automatically learn the optimal spatial pooling radius for location features, rather than relying on hand-tuned values?
• RQ4 How does the integration of location context affect performance on fine-grained, location-specific visual concepts?
• RQ5 To what extent does the model’s performance generalize across diverse geographic and semantic concepts?

Key findings

• The proposed method achieves a 7% relative improvement in mean average precision (from 36.82% to 43.78%) compared to a baseline image-only model.
• The best-performing model uses all five location features with a 256-dimensional representation and 10 replicas of the radius learning layer (256/- RL10), yielding 43.78% mean AP.
• Location-specific concepts like 'disneyland', 'casino', and 'alcatraz' show the largest performance gains, indicating strong localization effects.
• The model learns distinct pooling radii for different concepts: 'coast' and 'foliage' converge to 3 or fewer radii, while 'building' uses a broader range, reflecting context-dependent spatial scales.
• Adding a post-connection layer leads to overfitting, so the 256/- RL10 configuration without it is selected as the final model.
• The YFCC100M-GEO100 dataset, containing 88,986 geotagged images across 100 classes, is publicly released to support future research in location-aware vision.