[Paper Review] Using Social Media Images for Building Function Classification

Eike Jens Hoffmann, Karam Abdulahhad+1 more|arXiv (Cornell University)|Feb 15, 2022

Human Mobility and Location-Based Analysis4 citations

TL;DR

This paper proposes a scalable, content-first filtering pipeline to extract geotagged, ground-level social media images from Flickr for building function classification. By leveraging OSM building footprints and line-of-sight analysis, the method yields a weakly labeled dataset of 28M images across 42 cities, achieving an F1-score of 0.51 on a 3-class classification task, with performance limited primarily by OSM label quality rather than model architecture.

ABSTRACT

Urban land use on a building instance level is crucial geo-information for many applications, yet difficult to obtain. An intuitive approach to close this gap is predicting building functions from ground level imagery. Social media image platforms contain billions of images, with a large variety of motifs including but not limited to street perspectives. To cope with this issue this study proposes a filtering pipeline to yield high quality, ground level imagery from large social media image datasets. The pipeline ensures that all resulting images have full and valid geotags with a compass direction to relate image content and spatial objects from maps. We analyze our method on a culturally diverse social media dataset from Flickr with more than 28 million images from 42 cities around the world. The obtained dataset is then evaluated in a context of 3-classes building function classification task. The three building classes that are considered in this study are: commercial, residential, and other. Fine-tuned state-of-the-art architectures yield F1-scores of up to 0.51 on the filtered images. Our analysis shows that the performance is highly limited by the quality of the labels obtained from OpenStreetMap, as the metrics increase by 0.2 if only human validated labels are considered. Therefore, we consider these labels to be weak and publish the resulting images from our pipeline together with the buildings they are showing as a weakly labeled dataset.

Motivation & Objective

• To address the challenge of obtaining fine-grained urban land use data at the building level, especially for rapidly developing cities.
• To overcome the limitations of traditional aerial/satellite imagery by leveraging ground-level views from social media platforms.
• To develop a scalable, automated filtering pipeline that ensures valid geotags and compass orientation for social media images.
• To create a large-scale, weakly labeled benchmark dataset for building function classification using OSM labels.
• To evaluate the impact of OSM label quality on downstream classification performance and demonstrate the dataset's utility.

Proposed method

• A five-step filtering pipeline processes social media images to retain only those with valid geotags, compass direction, and line-of-sight to a building.
• Images are matched to OSM building footprints using a line-of-sight algorithm based on GPS coordinates, focal length, and compass heading from EXIF metadata.
• A cosine similarity-based image retrieval method uses a seed dataset to identify relevant ground-level views of buildings.
• Object detection is applied to identify buildings in images, and only the closest building in the field of view is assigned to the image.
• The pipeline is designed for scalability with only four hyperparameters, balancing runtime and output volume.
• Labels are derived from OSM’s building function tags, treating them as weak supervision due to known quality limitations.

Experimental results

Research questions

• RQ1 Can social media images with valid geotags and compass orientation be systematically filtered and linked to specific buildings for urban land use classification?
• RQ2 How effective is a weakly labeled dataset derived from OSM building functions for classifying building functions using deep learning?
• RQ3 To what extent does the quality of OSM labels limit the performance of building function classification models?
• RQ4 Can a content-first filtering pipeline scale to billions of social media images while preserving spatial accuracy?
• RQ5 How does performance vary when using human-validated labels versus OSM-derived labels in the same classification task?

Key findings

• The proposed filtering pipeline successfully extracted 28 million geotagged, ground-level images from Flickr across 42 global cities with valid compass orientation and spatial alignment to OSM buildings.
• Fine-tuned state-of-the-art models achieved an F1-score of 0.51 on the 3-class building function classification task (commercial, residential, other) using OSM-labeled data.
• When restricted to only human-validated labels, the F1-score increased by 0.20 to 0.71, indicating that OSM label quality is the primary performance bottleneck.
• The dataset is published as a benchmark with weak labels from OSM and a high-quality subset with human-validated labels from three independent raters.
• The study confirms that social media images are a viable, scalable source for urban land use analysis, provided that metadata quality and label reliability are carefully managed.
• The pipeline’s performance is robust across diverse urban environments and supports future extensions to multi-label or fine-grained classification schemes.