[论文解读] PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models

Valerio Marsocci, Yuru Jia+13 more|arXiv (Cornell University)|Dec 5, 2024

Geological Modeling and Analysis被引用 5

一句话总结

PANGAEA 引入了全球性、多样化的基准协议用于 Geospatial Foundation Models (GFMs)，评估若干 GFMs 相对于监督基线，并提供用于可重复、可扩展基准测试的开源代码。

ABSTRACT

Geospatial Foundation Models (GFMs) have emerged as powerful tools for extracting representations from Earth observation data, but their evaluation remains inconsistent and narrow. Existing works often evaluate on suboptimal downstream datasets and tasks, that are often too easy or too narrow, limiting the usefulness of the evaluations to assess the real-world applicability of GFMs. Additionally, there is a distinct lack of diversity in current evaluation protocols, which fail to account for the multiplicity of image resolutions, sensor types, and temporalities, which further complicates the assessment of GFM performance. In particular, most existing benchmarks are geographically biased towards North America and Europe, questioning the global applicability of GFMs. To overcome these challenges, we introduce PANGAEA, a standardized evaluation protocol that covers a diverse set of datasets, tasks, resolutions, sensor modalities, and temporalities. It establishes a robust and widely applicable benchmark for GFMs. We evaluate the most popular GFMs openly available on this benchmark and analyze their performance across several domains. In particular, we compare these models to supervised baselines (e.g. UNet and vanilla ViT), and assess their effectiveness when faced with limited labeled data. Our findings highlight the limitations of GFMs, under different scenarios, showing that they do not consistently outperform supervised models. PANGAEA is designed to be highly extensible, allowing for the seamless inclusion of new datasets, models, and tasks in future research. By releasing the evaluation code and benchmark, we aim to enable other researchers to replicate our experiments and build upon our work, fostering a more principled evaluation protocol for large pre-trained geospatial models. The code is available at https://github.com/VMarsocci/pangaea-bench.

研究动机与目标

• 激励对 GFMs 的稳健评估，超越狭窄的下游任务和地理偏见数据集。
• 建立覆盖城市、农业、海洋和森林环境的多域多样基准。
• 在多样的传感器、分辨率和时态条件下，评估泛化能力、数据效率及相对于监督基线的性能。
• 通过发布代码和模块化基准框架，促进可重复性和可扩展性。

提出的方法

• 策划一组跨领域、模态、时态和地理分布的多样化遥感数据集。
• 包含密集预测任务（语义分割、变化检测、回归），同时排除简单的补丁级分类和目标检测。
• 在不同训练条件下评估多种开源 GFM，包含自监督和监督基线（full vs limited labels）。
• 分析预训练数据特征（光谱丰富性、空间分辨率）以及下游任务/时态对 GFM 性能的影响。
• 提供可扩展的基准框架，支持添加新数据集、模型和任务，并附带开放评测代码。

Figure 1: Normalized performance comparison of different models across various datasets and training conditions. The y-axis represents the normalized performance across the 11 PANGAEA’s datasets, where the best-performing model for each dataset is assigned a value of 1 and the worst-performing model

实验结果

研究问题

• RQ1 GFMs 是否能够在多样化的下游领域和任务上有效泛化？
• RQ2 GFMs 在不同感知模态和时态设置下，是否始终优于监督基线？
• RQ3 预训练数据特征和标签可用性如何影响 GFM 的下游性能？
• RQ4 在任务和架构之间，微调与冻结编码器是否存在明显优势？

主要发现

• GFMs 在大多数任务上表现良好，但并非始终优于监督基线。
• 具有更丰富光谱信息或更高空间分辨率的预训练数据集，往往提升需要这些特征的下游任务的性能。
• 在有限标签设置下，一些 GFMs（如 CROMA）可以超过某些基线，但并非普遍现象。
• 对某些情况而言，微调可以提升性能，但并非普遍优于冻结编码器。

Figure 2: PANGAEA aims for robust evaluation across diverse downstream datasets and applications.