[Paper Review] A Practical Probabilistic Benchmark for AI Weather Models

Noah Brenowitz, Yair Cohen+7 more|arXiv (Cornell University)|Jan 27, 2024

Meteorological Phenomena and Simulations5 citations

TL;DR

The paper introduces lagged ensemble forecasts (LEF) as a parameter-free probabilistic benchmark to fairly compare AI weather models against an operational baseline, revealing that GraphCast and Pangu tie on probabilistic skill while long-lead-time training can harm ensemble calibration.

ABSTRACT

Since the weather is chaotic, forecasts aim to predict the distribution of future states rather than make a single prediction. Recently, multiple data driven weather models have emerged claiming breakthroughs in skill. However, these have mostly been benchmarked using deterministic skill scores, and little is known about their probabilistic skill. Unfortunately, it is hard to fairly compare AI weather models in a probabilistic sense, since variations in choice of ensemble initialization, definition of state, and noise injection methodology become confounding. Moreover, even obtaining ensemble forecast baselines is a substantial engineering challenge given the data volumes involved. We sidestep both problems by applying a decades-old idea -- lagged ensembles -- whereby an ensemble can be constructed from a moderately-sized library of deterministic forecasts. This allows the first parameter-free intercomparison of leading AI weather models' probabilistic skill against an operational baseline. The results reveal that two leading AI weather models, i.e. GraphCast and Pangu, are tied on the probabilistic CRPS metric even though the former outperforms the latter in deterministic scoring. We also reveal how multiple time-step loss functions, which many data-driven weather models have employed, are counter-productive: they improve deterministic metrics at the cost of increased dissipation, deteriorating probabilistic skill. This is confirmed through ablations applied to a spherical Fourier Neural Operator (SFNO) approach to AI weather forecasting. Separate SFNO ablations modulating effective resolution reveal it has a useful effect on ensemble dispersion relevant to achieving good ensemble calibration. We hope these and forthcoming insights from lagged ensembles can help guide the development of AI weather forecasts and have thus shared the diagnostic code.

Motivation & Objective

• Motivate the need for probabilistic evaluation of AI weather models beyond deterministic metrics.
• Propose lagged ensemble forecasts (LEF) as a parameter-free, scalable intercomparison framework.
• Enable fair comparisons across AI and traditional NWP models with minimized data and tuning requirements.
• Assess how long-lead-time training and effective resolution affect probabilistic skill and ensemble dispersion.

Proposed method

• Construct a 2-day centered lagged ensemble (M=4, h=12h, 9 members) from deterministic hindcasts.
• Evaluate probabilistic skill using CRPS, ensemble RMSE (eRMSE), deterministic RMSE (dRMSE), and spread vs. eRMSE calibration.
• Compare AI models (GraphCast, Pangu) against IFS HRES baseline using LEF.
• Perform ablations on SFNO to study long-lead-time autoregressive training effects on dispersion and skill.
• Investigate the impact of effective resolution (scale-factor) on ensemble dispersion.
• Provide diagnostic code and reproducible scoring workflow (github.com/NVIDIA/earth2mip).

Figure 1: Overview of lagged ensemble forecasting. Each ensemble member (color) is initialized at a different initial time (dots). The true time series (-) and the lagged ensemble average (- -) are also shown.

Experimental results

Research questions

• RQ1 Can lagged ensembles (LEF) serve as a meaningful probabilistic benchmark for AI weather models relative to operational baselines?
• RQ2 How do leading data-driven models (GraphCast, Pangu) compare probabilistically to IFS when evaluated with LEF?
• RQ3 What is the effect of multi-step, long-lead-time fine-tuning on probabilistic skill and ensemble dispersion?
• RQ4 How does changing the effective resolution of a data-driven model influence ensemble dispersion and calibration?

Key findings

• LEF provides a meaningful probabilistic proxy that correlates with the operational ensemble CRPS, enabling fair intercomparison.
• GraphCast and Pangu are virtually tied on probabilistic CRPS under LEF, even though GraphCast may outperform deterministically.
• Long-lead-time autoregressive fine-tuning reduces ensemble spread, improving deterministic RMSE but harming probabilistic skill and calibration.
• Lower effective resolution in SFNO increases ensemble dispersion (spread) for several fields, with modest or no clear gains in probabilistic skill.
• Using LEF with uniform weights reveals dispersion/skill dynamics and highlights dissipation-like effects from certain long-lead-time losses.
• Lagged-ensemble scoring is a practical, scalable benchmark that can guide AI weather forecast development and evaluation.

Figure 2: Comparison between the global mean CRPS of $\mathrm{5}\mathrm{0}\mathrm{0}$ hpa height forecasts from a lagged ensemble and the IFS operational ensemble at a lead time of 5 days. Each dot shows a different initial time. The units of each axes are \unit m^2/s^2