[Paper Review] Aardvark weather: end-to-end data-driven weather forecasting

Anna Vaughan, Stratis Markou+9 more|arXiv (Cornell University)|Mar 30, 2024

Hydrological Forecasting Using AI8 citations

TL;DR

Aardvark Weather presents the first end-to-end data-driven weather forecasting system that maps raw observations directly to forecasts, covering initial state estimation, global forecasting, and downstream station forecasts without external NWP input.

ABSTRACT

This resource contains the key source code, some sample data and the trained models from the paper: "End-to-end data-driven weather forecasting."

Motivation & Objective

• Motivate replacing the full NWP pipeline with a data-driven approach for end-to-end weather forecasting.
• Develop an end-to-end model that ingests raw observational data and outputs both global gridded forecasts and local station forecasts.
• Demonstrate skillful forecasts at 5–7 day lead times and show potential computational efficiency gains over traditional NWP.
• Enable training with reanalysis data pretraining and fine-tuning on observational data to address data scarcity in some regions.

Proposed method

• Propose a three-module neural process-based architecture (ConvCNP) with encoder, processor, and decoder to handle on-grid and off-grid data and missing values.
• Use UNet backbones for each module to enable scalable end-to-end learning.
• Encode observational inputs from in-situ and remote sensing sources into a gridded initial state, autoregressively forecast via a processor, and decode to local station forecasts.
• Train with ERA5 reanalysis data for pretraining, then fine-tune using 2007–2022 real observations to improve initial state and forecasts.
• Leverage SetConv layers to interpolate off-the-grid data and map gridded forecasts to arbitrary station locations, incorporating topographic information.

Figure 1 : Illustration of the complete end-to-end weather prediction pipeline. First, in the atmospheric state estimation stage (green), observed data from a range of sources including remote sensing platforms, marine observation platforms (e.g. ships) and in-situ land-based observation stations (w

Experimental results

Research questions

• RQ1 Can an end-to-end data-driven model generate both global and local forecasts without input from existing NWP systems?
• RQ2 What is the predictive skill of the end-to-end model for 24 global variables at multiple pressure levels and for local station forecasts?
• RQ3 How does the end-to-end model compare to baselines like persistence, climatology, and IFS-HRES in terms of accuracy and computational efficiency?
• RQ4 To what extent can pretraining on ERA5 reanalysis data improve performance when fine-tuned on observational data?
• RQ5 How well does the model handle off-the-grid data and missing observations across diverse geographical regions?

Key findings

• The model delivers skillful forecasts for 24 variables at multiple levels on a 1.41° grid and 24h temporal resolution, with five to seven day lead times.
• Global initial state estimation from raw observations achieves errors well below climatology and competitive with, but not exceeding, IFS-HRES in some metrics.
• For local station forecasts, Aardvark outperforms nearest-neighbor IFS-HRES baselines at multiple lead times for T2M, MSLP, and U10.
• The end-to-end system operates with substantially lower computational resources than traditional NWP baselines and can be trained on a small set of GPUs.
• Pretraining on ERA5 improves the usefulness of the encoder when fine-tuning on sparser observational data, addressing data scarcity in certain regions.