[论文解读] GeoLLM: Extracting Geospatial Knowledge from Large Language Models

Rohin Manvi, Samar Khanna+4 more|arXiv (Cornell University)|Oct 10, 2023

Geographic Information Systems Studies被引用 28

一句话总结

GeoLLM 在 prompts 中加入来自 OpenStreetMap 的数据以提取地理空间知识，对人口密度与资产财富估计等任务相比基线取得显著提升，在卫星数据基准线上表现接近或超过。

ABSTRACT

The application of machine learning (ML) in a range of geospatial tasks is increasingly common but often relies on globally available covariates such as satellite imagery that can either be expensive or lack predictive power. Here we explore the question of whether the vast amounts of knowledge found in Internet language corpora, now compressed within large language models (LLMs), can be leveraged for geospatial prediction tasks. We first demonstrate that LLMs embed remarkable spatial information about locations, but naively querying LLMs using geographic coordinates alone is ineffective in predicting key indicators like population density. We then present GeoLLM, a novel method that can effectively extract geospatial knowledge from LLMs with auxiliary map data from OpenStreetMap. We demonstrate the utility of our approach across multiple tasks of central interest to the international community, including the measurement of population density and economic livelihoods. Across these tasks, our method demonstrates a 70% improvement in performance (measured using Pearson's $r^2$) relative to baselines that use nearest neighbors or use information directly from the prompt, and performance equal to or exceeding satellite-based benchmarks in the literature. With GeoLLM, we observe that GPT-3.5 outperforms Llama 2 and RoBERTa by 19% and 51% respectively, suggesting that the performance of our method scales well with the size of the model and its pretraining dataset. Our experiments reveal that LLMs are remarkably sample-efficient, rich in geospatial information, and robust across the globe. Crucially, GeoLLM shows promise in mitigating the limitations of existing geospatial covariates and complementing them well. Code is available on the project website: https://rohinmanvi.github.io/GeoLLM

研究动机与目标

• 证明 LLMs 包含可用于预测任务的丰富地理空间知识。
• 证明用地图派生的上下文来增强提示能够改善对地理空间信息的提取。
• 在多样化的地理空间任务中，将微调的 LLM 与传统基线和卫星基准进行比较。

提出的方法

• 构建将坐标与地图派生上下文（来自 OpenStreetMap 的地址及附近地点）配对的提示。
• 在这些提示上对编码器/解码器 LLMs（RoBERTa、Llama 2、GPT-3.5）进行微调，以预测地理空间目标（四舍五入到 0.0–9.9 量表）。
• 使用最小可行的地理提示并对组件进行消融以评估地图数据的贡献（地址、附近地点）。
• 应用 LoRA/QLoRA 的微调以减少训练参数并保持权重知识。
• 通过在多个全球数据集上预测ground truth的 Pearson r^2 进行评估；与 k-NN、XGBoost 等基线进行比较。

Figure 2: Plots of absolute error comparing the best baselines and GPT-3.5 on tasks from each source with 1,000 samples. We also provide high-resolution plots from various locations around the world for the population density task from WorldPop. We show that GeoLLM not only outperforms baselines on

实验结果

研究问题

• RQ1 LLMs 是否编码了超越仅凭坐标就可用的地理空间知识？
• RQ2 地图派生的提示上下文是否能改善 LLMs 的地理空间预测？
• RQ3 不同模型规模与预训练数据如何影响地理知识提取？
• RQ4 GeoLLM 相较于卫星基变量和传统基线在不同地区的性能如何？

主要发现

• LLMs 含有大量地理空间知识；仅用坐标提示往往不足以发挥潜力。
• 用地址和来自 OpenStreetMap 的邻近地点来增强提示显著提升性能；例如，GPT-3.5 相较于仅坐标提示在人口密度上的 r^2 提升了 3.3 倍。
• GPT-3.5、Llama 2 与 RoBERTa 在基线（k-NN、XGBoost 变体）上的改进分别达到最大 70%、43%、13%，在大多数任务中 GPT-3.5 通常表现最好。
• GPT-3.5 在多项任务与样本规模（100、1,000、10,000）下表现始终强劲，常常达到或超过卫星基准。
• 该方法展现出地理一致性和样本效率；较大模型配合更大的预训练数据往往表现更好，显示 GeoLLM 的可扩展性。