[Paper Review] AI Foundation Models for Weather and Climate: Applications, Design, and Implementation

S. Karthik Mukkavilli, Daniel Civitarese+13 more|arXiv (Cornell University)|Sep 19, 2023

Meteorological Phenomena and Simulations14 citations

TL;DR

The paper surveys the rise of AI foundation models for weather and climate, outlining applications, design choices, and implementation considerations toward a generalizable weather foundation model.

ABSTRACT

Machine learning and deep learning methods have been widely explored in understanding the chaotic behavior of the atmosphere and furthering weather forecasting. There has been increasing interest from technology companies, government institutions, and meteorological agencies in building digital twins of the Earth. Recent approaches using transformers, physics-informed machine learning, and graph neural networks have demonstrated state-of-the-art performance on relatively narrow spatiotemporal scales and specific tasks. With the recent success of generative artificial intelligence (AI) using pre-trained transformers for language modeling and vision with prompt engineering and fine-tuning, we are now moving towards generalizable AI. In particular, we are witnessing the rise of AI foundation models that can perform competitively on multiple domain-specific downstream tasks. Despite this progress, we are still in the nascent stages of a generalizable AI model for global Earth system models, regional climate models, and mesoscale weather models. Here, we review current state-of-the-art AI approaches, primarily from transformer and operator learning literature in the context of meteorology. We provide our perspective on criteria for success towards a family of foundation models for nowcasting and forecasting weather and climate predictions. We also discuss how such models can perform competitively on downstream tasks such as downscaling (super-resolution), identifying conditions conducive to the occurrence of wildfires, and predicting consequential meteorological phenomena across various spatiotemporal scales such as hurricanes and atmospheric rivers. In particular, we examine current AI methodologies and contend they have matured enough to design and implement a weather foundation model.

Motivation & Objective

• Motivate the development of foundation models for global and regional weather and climate problems.
• Summarize state-of-the-art AI approaches (transformers, graph networks, physics-informed ML) in meteorology.
• Discuss design criteria, data requirements, and evaluation considerations for weather foundation models.
• Identify downstream tasks where foundation models can excel (nowcasting, downscaling, parameterization, data assimilation).
• Outline roadmap and criteria for success of a weather foundation model family.

Proposed method

• Review current state-of-the-art AI methods in meteorology (transformers, graph neural networks, operator learning).
• Conceptualize the encoder–decoder foundation model structure and fine-tuning paradigm for downstream tasks.
• Discuss pretraining data needs and loss-function considerations for self-supervised learning in weather/climate contexts.
• Analyze design tradeoffs between backbones (transformers vs. graph-based) and multi-scale data representations.
• Propose evaluation and diagnostics criteria aligned with meteorological realism and downstream utility.

Experimental results

Research questions

• RQ1 What are the key downstream tasks where a weather and climate foundation model could be effectively applied?
• RQ2 What design criteria and data requirements are needed to build a generalizable weather foundation model?
• RQ3 How can foundation models improve tasks such as downscaling, nowcasting, data assimilation, and parameterization?
• RQ4 What are the challenges and limitations of applying FM approaches to weather/climate domains, and how might they be mitigated?
• RQ5 What roadmap and milestones are necessary for long-term rollout of weather foundation models?

Key findings

• Foundation models can potentially improve accuracy and efficiency across weather/climate variables and timescales.
• Pretraining on large, diverse datasets reduces labeled data needs for fine-tuning on downstream tasks.
• Fine-tuning with lightweight task-specific decoders enables versatile, multi-task deployment.
• Current AI emulators and models achieve comparable performance to some NWP systems but face grain-size, lead-time, and generalization limitations.
• Downscaling, hazard detection, and climate applications could benefit from FM approaches through better generalization and data efficiency.
• Notable examples (e.g., FourCastNet, PanguWeather, GraphCast) illustrate rapid progress toward scalable, high-resolution modeling.