[Paper Review] AIFS-CRPS: Ensemble forecasting using a model trained with a loss function based on the Continuous Ranked Probability Score

Simon Lang, Mihai Alexe+16 more|arXiv (Cornell University)|Dec 20, 2024

Forecasting Techniques and Applications6 citations

TL;DR

AIFS-CRPS trains a machine-learned ensemble forecast model using a probabilistic afCRPS loss to produce exchangeable ensemble members, achieving competitive or superior medium-range and subseasonal skill compared to the physics-based IFS ensemble.

ABSTRACT

Over the last three decades, ensemble forecasts have become an integral part of forecasting the weather. They provide users with more complete information than single forecasts as they permit to estimate the probability of weather events by representing the sources of uncertainties and accounting for the day-to-day variability of error growth in the atmosphere. This paper presents a novel approach to obtain a weather forecast model for ensemble forecasting with machine-learning. AIFS-CRPS is a variant of the Artificial Intelligence Forecasting System (AIFS) developed at ECMWF. Its loss function is based on a proper score, the Continuous Ranked Probability Score (CRPS). For the loss, the almost fair CRPS is introduced because it approximately removes the bias in the score due to finite ensemble size yet avoids a degeneracy of the fair CRPS. The trained model is stochastic and can generate as many exchangeable members as desired and computationally feasible in inference. For medium-range forecasts AIFS-CRPS outperforms the physics-based Integrated Forecasting System (IFS) ensemble for the majority of variables and lead times. For subseasonal forecasts, AIFS-CRPS outperforms the IFS ensemble before calibration and is competitive with the IFS ensemble when forecasts are evaluated as anomalies to remove the influence of model biases.

Motivation & Objective

• Motivate the need for probabilistic, machine-learned ensemble forecasts in weather prediction.
• Develop an ensemble forecasting model (AIFS-CRPS) trained with a continuous ranked probability score variant to better represent uncertainty.
• Assess the performance of AIFS-CRPS against the ECMWF IFS ensemble across medium-range and subseasonal forecasts.

Proposed method

• Extend the AIFS architecture with probabilistic training using a ground-truth deterministic state (ERA5 or IFS analysis).
• Use 16 processor layers with a transformer-based encoder/decoder within an octahedral reduced Gaussian grid framework.
• Train with almost fair CRPS (afCRPS) to balance bias correction for finite ensemble size and avoid score degeneracy.
• Represent ensemble members via independent Gaussian noise inputs processed per member, with ensemble sharding across GPUs for scalability.
• Employ auto-regressive rollout during training and inference to simulate multi-step forecasts up to 72 hours or more.
• Utilize a four-stage training scheme with staged rollout lengths and cosine learning-rate schedules, and a loss based on afCRPS.”

Figure 1: Probabilistic training of AIFS-CRPS. A small ensemble of atmospheric states is propagated forward in time using separate model instances (that share the same weights). With ensemble sharding (see section 2.4 ), the ensemble forecasts are then gathered across all participating GPU devices u

Experimental results

Research questions

• RQ1 Does AIFS-CRPS provide reliable probabilistic forecasts by learning to represent model uncertainty through a CRPS-based loss?
• RQ2 How does AIFS-CRPS perform relative to the physics-based IFS ensemble for medium-range forecasts across multiple variables and lead times?
• RQ3 Can AIFS-CRPS maintain ensemble variability without excessive smoothing or over-dispersion, particularly for extreme events?
• RQ4 What is the impact of grid resolution, ensemble size, and pressure-scaling in the loss on forecast skill and spread?
• RQ5 Is AIFS-CRPS competitive for subseasonal forecasts when evaluated as raw fields and as anomalies?

Key findings

• AIFS-CRPS ensembles retain realistic variability over lead times, avoiding the smoothing seen in MSE-trained models.
• Compared with the IFS ensemble, AIFS-CRPS is competitive or superior for many medium-range variables, with improvements in CRPS and RMSE in several regions.
• Higher-resolution AIFS-CRPS (N320) generally yields greater skill than the lower-resolution setup (O96) for numerous surface variables, though some upper-atmosphere variables show degradation due to pressure scaling choices.
• AIFS-CRPS demonstrates subseasonal skill improvements relative to IFS for many surface and tropospheric parameters at 2–3 week lead times, with anomaly-based assessments showing persistent benefits.
• The method delivers better MJO-related forecast characteristics and ensemble behavior than IFS in several metrics, including correlations and RMSE of the ensemble mean.
• Overall, AIFS-CRPS produces a larger and sometimes better-calibrated ensemble spread in the extra-tropics, and can outperform the IFS ensemble in many cases while maintaining reasonable ensemble statistics.

(a)