[Paper Review] GeoLLM: Extracting Geospatial Knowledge from Large Language Models

Rohin Manvi, Samar Khanna+4 more|arXiv (Cornell University)|Oct 10, 2023

Geographic Information Systems StudiesSocial Sciences28 citations

TL;DR

GeoLLM fine-tunes LLMs on prompts augmented with OpenStreetMap data to extract geospatial knowledge, achieving substantial improvements over baselines in tasks like population density and asset wealth estimation, and showing performance on par with or exceeding satellite-based benchmarks.

ABSTRACT

The application of machine learning (ML) in a range of geospatial tasks is increasingly common but often relies on globally available covariates such as satellite imagery that can either be expensive or lack predictive power. Here we explore the question of whether the vast amounts of knowledge found in Internet language corpora, now compressed within large language models (LLMs), can be leveraged for geospatial prediction tasks. We first demonstrate that LLMs embed remarkable spatial information about locations, but naively querying LLMs using geographic coordinates alone is ineffective in predicting key indicators like population density. We then present GeoLLM, a novel method that can effectively extract geospatial knowledge from LLMs with auxiliary map data from OpenStreetMap. We demonstrate the utility of our approach across multiple tasks of central interest to the international community, including the measurement of population density and economic livelihoods. Across these tasks, our method demonstrates a 70% improvement in performance (measured using Pearson's $r^2$) relative to baselines that use nearest neighbors or use information directly from the prompt, and performance equal to or exceeding satellite-based benchmarks in the literature. With GeoLLM, we observe that GPT-3.5 outperforms Llama 2 and RoBERTa by 19% and 51% respectively, suggesting that the performance of our method scales well with the size of the model and its pretraining dataset. Our experiments reveal that LLMs are remarkably sample-efficient, rich in geospatial information, and robust across the globe. Crucially, GeoLLM shows promise in mitigating the limitations of existing geospatial covariates and complementing them well. Code is available on the project website: https://rohinmanvi.github.io/GeoLLM

Motivation & Objective

• Demonstrate that LLMs contain rich geospatial knowledge that can be harnessed for prediction tasks.
• Show that augmenting prompts with map-derived context improves extraction of geospatial information.
• Compare fine-tuned LLMs to traditional baselines and satellite-based benchmarks across diverse geospatial tasks.

Proposed method

• Construct prompts that pair coordinates with map-derived context (address and nearby places from OpenStreetMap).
• Fine-tune encoder/decoder LLMs (RoBERTa, Llama 2, GPT-3.5) on these prompts to predict a geospatial target (rounded to 0.0–9.9 scale).
• Use minimum viable geospatial prompts and ablate components to assess contribution of map data (address, nearby places).
• Apply fine-tuning with LoRA/QLoRA to reduce parameters trained and maintain weight knowledge.
• Evaluate by predicting Pearson r^2 against ground truth across multiple global datasets; compare to baselines like k-NN and XGBoost.

Figure 2: Plots of absolute error comparing the best baselines and GPT-3.5 on tasks from each source with 1,000 samples. We also provide high-resolution plots from various locations around the world for the population density task from WorldPop. We show that GeoLLM not only outperforms baselines on

Experimental results

Research questions

• RQ1 Do LLMs encode usable geospatial knowledge beyond what is accessible via coordinates alone?
• RQ2 Can map-derived prompt context improve geospatial predictions from LLMs?
• RQ3 How do different model sizes and pretraining data affect geospatial knowledge extraction?
• RQ4 How does GeoLLM perform relative to satellite-based covariates and traditional baselines across diverse geographies?

Key findings

• LLMs contain substantial geospatial knowledge; prompting with coordinates alone is often insufficient.
• Augmenting prompts with address and nearby places from OpenStreetMap markedly improves performance; for example, GPT-3.5 gains 3.3x in r^2 over coordinate-only prompts for population density.
• GPT-3.5, Llama 2, and RoBERTa achieve up to 70%, 43%, and 13% improvements over baselines (k-NN, XGBoost variants) respectively; GPT-3.5 generally outperforms others across tasks.
• Performance of GPT-3.5 is consistently strong across tasks and sample sizes (100, 1,000, 10,000), often matching or exceeding satellite-based benchmarks.
• The method shows geographic consistency and sample efficiency; larger models with bigger pretraining data tend to perform better, suggesting scalability of GeoLLM.