[论文解读] A Practical Probabilistic Benchmark for AI Weather Models

Noah Brenowitz, Yair Cohen+7 more|arXiv (Cornell University)|Jan 27, 2024

Meteorological Phenomena and Simulations被引用 5

一句话总结

The paper introduces lagged ensemble forecasts (LEF) as a parameter-free probabilistic benchmark to fairly compare AI weather models against an operational baseline, revealing that GraphCast and Pangu tie on probabilistic skill while long-lead-time training can harm ensemble calibration.

ABSTRACT

Since the weather is chaotic, forecasts aim to predict the distribution of future states rather than make a single prediction. Recently, multiple data driven weather models have emerged claiming breakthroughs in skill. However, these have mostly been benchmarked using deterministic skill scores, and little is known about their probabilistic skill. Unfortunately, it is hard to fairly compare AI weather models in a probabilistic sense, since variations in choice of ensemble initialization, definition of state, and noise injection methodology become confounding. Moreover, even obtaining ensemble forecast baselines is a substantial engineering challenge given the data volumes involved. We sidestep both problems by applying a decades-old idea -- lagged ensembles -- whereby an ensemble can be constructed from a moderately-sized library of deterministic forecasts. This allows the first parameter-free intercomparison of leading AI weather models' probabilistic skill against an operational baseline. The results reveal that two leading AI weather models, i.e. GraphCast and Pangu, are tied on the probabilistic CRPS metric even though the former outperforms the latter in deterministic scoring. We also reveal how multiple time-step loss functions, which many data-driven weather models have employed, are counter-productive: they improve deterministic metrics at the cost of increased dissipation, deteriorating probabilistic skill. This is confirmed through ablations applied to a spherical Fourier Neural Operator (SFNO) approach to AI weather forecasting. Separate SFNO ablations modulating effective resolution reveal it has a useful effect on ensemble dispersion relevant to achieving good ensemble calibration. We hope these and forthcoming insights from lagged ensembles can help guide the development of AI weather forecasts and have thus shared the diagnostic code.

研究动机与目标

• 推动在超越确定性指标的基础上，对 AI 天气模型进行概率评估的需求。
• 提出 lagged ensemble forecasts (LEF) 作为一个无参数、可扩展的互比较框架。
• 在尽量减少数据与调优需求的情况下，实现 AI 与传统 NWP 模型之间的公平比较。
• 评估长期前置时间训练与有效分辨率如何影响概率技能与集合离散度。

提出的方法

• 从确定性逆预报构建一个双日中心的滞后集合 (M=4, h=12h, 9 members)。
• 使用 CRPS、ensemble RMSE (eRMSE)、deterministic RMSE (dRMSE) 以及 spread 与 eRMSE 的校准来评估概率技能。
• 使用 LEF 将 AI 模型 (GraphCast, Pangu) 与 IFS HRES 基线进行比较。
• 对 SFNO 进行消融研究，以探讨长期前置时间自回归训练对离散度与技能的影响。
• 研究有效分辨率（scale-factor）对集合离散度的影响。
• 提供诊断代码和可重复的评分工作流（github.com/NVIDIA/earth2mip）。

Figure 1: Overview of lagged ensemble forecasting. Each ensemble member (color) is initialized at a different initial time (dots). The true time series (-) and the lagged ensemble average (- -) are also shown.

实验结果

研究问题

• RQ1 滞后集合（LEF）是否可以作为相对于运营基线的有意义的 AI 天气模型概率基准？
• RQ2 在使用 LEF 评估时，领先的数据驱动模型（GraphCast、Pangu）在概率上与 IFS 的比较如何？
• RQ3 多步、长前置时间微调对概率技能与集合离散度的影响是什么？
• RQ4 改变数据驱动模型的有效分辨率如何影响集合离散度与校准？

主要发现

• LEF 提供了一个有意义的概率代理，与运营集合 CRPS 相关，从而实现公平的互比较。
• 在 LEF 下，GraphCast 与 Pangu 在概率 CRPS 上几乎并列，尽管 GraphCast 在确定性方面可能更优。
• 长期前置时间自回归微调降低了集合离散度，提升确定性 RMSE，但损害概率技能与校准。
• SFNO 的较低有效分辨率在若干场上增加集合离散度（spread），在概率技能方面的提升有限或无明显提升。
• 使用 LEF 与统一权重揭示了离散度/技能的动态，并凸显某些长期前置损失产生的类似耗散的效应。
• 滞后集合评分是一种实用、可扩展的基准，可以指导 AI 天气预报的发展与评估。

Figure 2: Comparison between the global mean CRPS of $\mathrm{5}\mathrm{0}\mathrm{0}$ hpa height forecasts from a lagged ensemble and the IFS operational ensemble at a lead time of 5 days. Each dot shows a different initial time. The units of each axes are \unit m^2/s^2