[Paper Review] On the Foundations of Earth and Climate Foundation Models

Xiao Xiang Zhu, Zhitong Xiong+8 more|arXiv (Cornell University)|May 7, 2024

Geological Studies and Exploration16 citations

TL;DR

Defines an ideal Earth and climate foundation model with eleven required features, surveys state-of-the-art, and outlines data and design directions to move toward unified, sustainable Earth system FMs.

ABSTRACT

Foundation models have enormous potential in advancing Earth and climate sciences, however, current approaches may not be optimal as they focus on a few basic features of a desirable Earth and climate foundation model. Crafting the ideal Earth foundation model, we define eleven features which would allow such a foundation model to be beneficial for any geoscientific downstream application in an environmental- and human-centric manner.We further shed light on the way forward to achieve the ideal model and to evaluate Earth foundation models. What comes after foundation models? Energy efficient adaptation, adversarial defenses, and interpretability are among the emerging directions.

Motivation & Objective

• Motivate the need for a unified Earth and climate foundation model that can handle diverse data modalities and downstream tasks.
• Define a concrete set of must-have and desirable features to guide FM design in EO, weather, and climate domains.
• Assess current representative Earth and climate FMs against these features and identify gaps and priorities for future work.
• Propose data construction and model-design principles to realize an ideal, carbon-minimized, physically consistent FM.
• Discuss evaluation benchmarks and progressing toward standardized benchmarking for Earth FMs.

Proposed method

• Proposes a framework and feature checklist (eleven features) for ideal Earth and climate foundation models.
• Reviews representative EO and climate FMs and their alignment with the proposed features.
• Outlines data-centric and model-centric pathways toward the ideal FM, including data diversity, geolocation, and multi-modal fusion.
• Suggests architectural and training strategies such as dynamic encoders, spatio-temporal modeling, and geographical mixture of experts.
• Discusses evaluation benchmarks and the need for standardized tasks across EO, weather, and climate modalities.

Experimental results

Research questions

• RQ1 What are the essential features that an ideal Earth and climate foundation model should possess?
• RQ2 How do current Earth and climate foundation models measure up against these features across EO, weather, and climate domains?
• RQ3 What data and architectural design strategies are needed to realize a comprehensive, scalable, and carbon-minimized Earth FM?
• RQ4 What framework and benchmarks are required to evaluate Earth FMs comprehensively across diverse tasks and modalities.

Key findings

• There is a gap between current models and the ideal FM, with many models addressing only two to three of the eight must-have features.
• EO foundation models have been more mature, often enabling geolocation embedding and time features, but climate FMs lag in high-resolution, multi-sensor integration and physical consistency.
• Representative EO and climate FMs vary in feature coverage, showing progress but not a unified, fully-featured FM.
• A forward path emphasizes comprehensive data curation, dynamic encoding, spatio-temporal multi-scale modeling, GeoMoE, continual pre-training, uncertainty quantification, physics alignment, and language alignment.
• Benchmarking needs are identified: data diversity, representative tasks, and inter-component interactions across EO, weather, and climate data.