[论文解读] On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence

Gengchen Mai, Weiming Huang+12 more|arXiv (Cornell University)|Apr 13, 2023

Data-Driven Disease Surveillance被引用 65

一句话总结

本论文系统评估现有基础模型（FMs）在地理空间领域的表现，显示文本仅任务在零-shot/少量-shot 设置中受益于LLMs，而多模态 GeoAI 任务仍然需要专门的任务模型；提出一个用于 GeoAI 的多模态基础模型框架，并讨论风险。

ABSTRACT

Large pre-trained models, also known as foundation models (FMs), are trained in a task-agnostic manner on large-scale data and can be adapted to a wide range of downstream tasks by fine-tuning, few-shot, or even zero-shot learning. Despite their successes in language and vision tasks, we have yet seen an attempt to develop foundation models for geospatial artificial intelligence (GeoAI). In this work, we explore the promises and challenges of developing multimodal foundation models for GeoAI. We first investigate the potential of many existing FMs by testing their performances on seven tasks across multiple geospatial subdomains including Geospatial Semantics, Health Geography, Urban Geography, and Remote Sensing. Our results indicate that on several geospatial tasks that only involve text modality such as toponym recognition, location description recognition, and US state-level/county-level dementia time series forecasting, these task-agnostic LLMs can outperform task-specific fully-supervised models in a zero-shot or few-shot learning setting. However, on other geospatial tasks, especially tasks that involve multiple data modalities (e.g., POI-based urban function classification, street view image-based urban noise intensity classification, and remote sensing image scene classification), existing foundation models still underperform task-specific models. Based on these observations, we propose that one of the major challenges of developing a FM for GeoAI is to address the multimodality nature of geospatial tasks. After discussing the distinct challenges of each geospatial data modality, we suggest the possibility of a multimodal foundation model which can reason over various types of geospatial data through geospatial alignments. We conclude this paper by discussing the unique risks and challenges to develop such a model for GeoAI.

研究动机与目标

• 评估现有基础模型在跨多个子领域（地理空间语义、健康地理、城市地理、遥感）上的地理空间任务表现。
• 识别在 GeoAI 任务中任务无关的基础模型的优势与局限性，特别是对多模态数据。
• 讨论挑战并勾画一个为 GeoAI 任务定制的多模态基础模型的愿景。
• 突出在开发与部署 GeoAI 基础模型时的风险和注意事项。

提出的方法

• 在四个领域的七个地理空间任务上基准测试若干预训练的基础模型（LLMs、视觉和多模态）。
• 将 FM 的表现与最先进的全监督、任务特定模型进行比较。
• 对于文本中心的任务使用零-shot 和少-shot 提示；对地形/语义任务实现带少-shot 示例的提示。
• 评估对地名识别、地点描述识别、痴呆症死亡时间序列预测（州与县级）、基于兴趣点的城市功能分类、街景图像的噪声强度分类，以及遥感图像场景分类的表现。
• 分析结果以识别模态特定的优势/劣势，以及模型规模和提示策略的影响。

(a) Persistence

实验结果

研究问题

• RQ1 现有基础模型在零-shot/少-shot 设置下是否能够匹配或超过任务特定模型在地理空间语义任务上的表现？
• RQ2 在健康地理、城市地理和遥感任务，尤其是多模态任务，FM 的表现是否出色？
• RQ3 将 FM 应用于多模态 GeoAI 数据的主要挑战是什么，如何通过多模态 GeoFM 框架来应对？
• RQ4 在开发与部署多模态 GeoAI 基础模型时会出现哪些风险？

主要发现

• 在文本专用地理空间任务（地名识别和地点描述识别）中，若干模型在零-shot/少-shot 设置下，LLMs 能超越特定监督基线；GPT-3 及相关模型在某些数据集上显示显著提升。
• 在多模态 GeoAI 任务（如基于POI的城市功能分类、街景图像的噪声强度分类、遥感图像场景分类）中，现有FM 的表现不如任务特定模型。
• GPT-3、InstructGPT 及某些 ChatGPT 变体在州级痴呆时序预测中表现强劲，在零-shot 设置下有时超越基于 ARIMA 的基线；GPT-2 家族通常落后于时序基线。
• 在州级痴呆预测中，InstructGPT 在多项指标上可能优于 ARIMA，而 GPT-2 模型显著落后；县级结果呈现类似趋势。
• 总体而言，多模态 GeoAI 仍是当前 FM 的关键挑战，促使需要一个具备地理空间对齐的多模态 GeoAI 基础模型。

(b) ARIMA