[Paper Review] Foundation Models for Weather and Climate Data Understanding: A Comprehensive Survey

Shengchao Chen, Guodong Long+3 more|arXiv (Cornell University)|Dec 5, 2023

Meteorological Phenomena and Simulations10 citations

TL;DR

This paper surveys state-of-the-art foundation models and task-specific models for weather and climate data, focusing on time series, spatio-temporal data, and text data, and discusses datasets, architectures, applications, challenges, and future directions.

ABSTRACT

As artificial intelligence (AI) continues to rapidly evolve, the realm of Earth and atmospheric sciences is increasingly adopting data-driven models, powered by progressive developments in deep learning (DL). Specifically, DL techniques are extensively utilized to decode the chaotic and nonlinear aspects of Earth systems, and to address climate challenges via understanding weather and climate data. Cutting-edge performance on specific tasks within narrower spatio-temporal scales has been achieved recently through DL. The rise of large models, specifically large language models (LLMs), has enabled fine-tuning processes that yield remarkable outcomes across various downstream tasks, thereby propelling the advancement of general AI. However, we are still navigating the initial stages of crafting general AI for weather and climate. In this survey, we offer an exhaustive, timely overview of state-of-the-art AI methodologies specifically engineered for weather and climate data, with a special focus on time series and text data. Our primary coverage encompasses four critical aspects: types of weather and climate data, principal model architectures, model scopes and applications, and datasets for weather and climate. Furthermore, in relation to the creation and application of foundation models for weather and climate data understanding, we delve into the field's prevailing challenges, offer crucial insights, and propose detailed avenues for future research. This comprehensive approach equips practitioners with the requisite knowledge to make substantial progress in this domain. Our survey encapsulates the most recent breakthroughs in research on large, data-driven models for weather and climate data understanding, emphasizing robust foundations, current advancements, practical applications, crucial resources, and prospective research opportunities.

Motivation & Objective

• Provide an up-to-date overview of large foundation models and task-specific models for weather and climate data.
• categorize models by architecture and data modality (time series, spatio-temporal, text).
• Summarize datasets, tools, and resources available for weather/climate AI research.
• Highlight challenges, opportunities, and design considerations for climate foundation models.

Proposed method

• Systematic literature review of foundation models and task-specific models applied to weather and climate data.
• Categorization of models by architecture (RNNs, Transformers, GANs, Diffusion, GNNs) and by data modality (time series, spatio-temporal, text).
• Discussion of data types, tasks, and representative applications in weather/climate science.
• Compilation of datasets, resources, and open-source tools relevant to the field.
• Articulation of design principles and future research directions for climate foundation models.

Experimental results

Research questions

• RQ1 What are the current foundation models and task-specific models used for weather and climate data understanding?
• RQ2 How are these models distributed across data modalities and architectural families?
• RQ3 What datasets and tools are available to support research in weather and climate AI?
• RQ4 What challenges and opportunities define the development of weather/climate foundation models?

Key findings

• The survey provides a comprehensive, contemporary overview of large-scale and task-specific models for weather and climate data across time series, spatio-temporal data, and text.
• It offers a structured categorization into climate foundation models and task-specific models, detailed by architectures such as RNNs, Transformers, GANs, Diffusion models, and GNNs.
• A rich compilation of datasets, open-source implementations, and resources is presented to aid researchers.
• The paper identifies challenges and outlines future research directions in data representation, multi-modal modeling, interpretability, generalizability, privacy, and continual learning.
• It provides design insights for building weather and climate foundation models, including data choices, representation, learning strategies, and evaluation schemes.