[Paper Review] PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models

Valerio Marsocci, Yuru Jia+13 more|arXiv (Cornell University)|Dec 5, 2024

Geological Modeling and Analysis5 citations

TL;DR

PANGAEA introduces a global, diverse benchmark protocol for Geospatial Foundation Models (GFMs), evaluates several GFMs against supervised baselines, and provides open-source code for reproducible, extensible benchmarking.

ABSTRACT

Geospatial Foundation Models (GFMs) have emerged as powerful tools for extracting representations from Earth observation data, but their evaluation remains inconsistent and narrow. Existing works often evaluate on suboptimal downstream datasets and tasks, that are often too easy or too narrow, limiting the usefulness of the evaluations to assess the real-world applicability of GFMs. Additionally, there is a distinct lack of diversity in current evaluation protocols, which fail to account for the multiplicity of image resolutions, sensor types, and temporalities, which further complicates the assessment of GFM performance. In particular, most existing benchmarks are geographically biased towards North America and Europe, questioning the global applicability of GFMs. To overcome these challenges, we introduce PANGAEA, a standardized evaluation protocol that covers a diverse set of datasets, tasks, resolutions, sensor modalities, and temporalities. It establishes a robust and widely applicable benchmark for GFMs. We evaluate the most popular GFMs openly available on this benchmark and analyze their performance across several domains. In particular, we compare these models to supervised baselines (e.g. UNet and vanilla ViT), and assess their effectiveness when faced with limited labeled data. Our findings highlight the limitations of GFMs, under different scenarios, showing that they do not consistently outperform supervised models. PANGAEA is designed to be highly extensible, allowing for the seamless inclusion of new datasets, models, and tasks in future research. By releasing the evaluation code and benchmark, we aim to enable other researchers to replicate our experiments and build upon our work, fostering a more principled evaluation protocol for large pre-trained geospatial models. The code is available at https://github.com/VMarsocci/pangaea-bench.

Motivation & Objective

• Motivate robust evaluation of GFMs beyond narrow downstream tasks and geo-biased datasets.
• Establish a diverse, multi-domain benchmark covering urban, agricultural, marine, and forest environments.
• Assess generalization, data efficiency, and performance relative to supervised baselines across varied sensors, resolutions, and temporality.
• Promote reproducibility and extensibility by releasing code and a modular benchmarking framework.

Proposed method

• Curate a diverse set of EO datasets spanning domains, modalities, temporality, and geography.
• Include dense-prediction tasks (semantic segmentation, change detection, regression) while excluding simple patch-level classification and object detection.
• Evaluate multiple open-source GFMs, including self-supervised and supervised baselines, under various training conditions (full vs limited labels).
• Analyze effects of pre-training data characteristics (spectral richness, spatial resolution) and downstream task/temporal alignment on GFM performance.
• Provide an extensible benchmark framework that supports adding new datasets, models, and tasks, with open evaluation code.

Figure 1: Normalized performance comparison of different models across various datasets and training conditions. The y-axis represents the normalized performance across the 11 PANGAEA’s datasets, where the best-performing model for each dataset is assigned a value of 1 and the worst-performing model

Experimental results

Research questions

• RQ1 Do GFMs generalize effectively across diverse downstream domains and tasks?
• RQ2 Do GFMs consistently outperform supervised baselines across varied sensing modalities and temporal settings?
• RQ3 How do pre-training data characteristics and label availability affect GFM downstream performance?
• RQ4 Is there a clear advantage to fine-tuning versus freezing encoders across tasks and architectures?

Key findings

• GFMs generally perform well across tasks but do not consistently outperform supervised baselines.
• Pre-training datasets with richer spectral information or higher spatial resolution tend to boost downstream performance for tasks requiring those features.
• In limited-label settings, some GFMs (e.g., CROMA) can surpass some baselines, but this is not universal.
• Fine-tuning improves performance in some cases but is not universally superior to freezing the encoder.

Figure 2: PANGAEA aims for robust evaluation across diverse downstream datasets and applications.