[Paper Review] A Machine Learning Outlook: Post-processing of Global Medium-range Forecasts

Shreya Agrawal, Robert W. Carver+8 more|arXiv (Cornell University)|Mar 28, 2023

Meteorological Phenomena and Simulations8 citations

TL;DR

The paper investigates non-linear neural network post-processing (ML-PP) of global medium-range forecasts, using 173 weather features globally to improve forecast accuracy up to 7 days ahead, with RMSE gains up to ~14% for some variables.

ABSTRACT

Post-processing typically takes the outputs of a Numerical Weather Prediction (NWP) model and applies linear statistical techniques to produce improve localized forecasts, by including additional observations, or determining systematic errors at a finer scale. In this pilot study, we investigate the benefits and challenges of using non-linear neural network (NN) based methods to post-process multiple weather features -- temperature, moisture, wind, geopotential height, precipitable water -- at 30 vertical levels, globally and at lead times up to 7 days. We show that we can achieve accuracy improvements of up to 12% (RMSE) in a field such as temperature at 850hPa for a 7 day forecast. However, we recognize the need to strengthen foundational work on objectively measuring a sharp and correct forecast. We discuss the challenges of using standard metrics such as root mean squared error (RMSE) or anomaly correlation coefficient (ACC) as we move from linear statistical models to more complex non-linear machine learning approaches for post-processing global weather forecasts.

Motivation & Objective

• Assess whether non-linear neural networks can improve post-processing of global NWP outputs over traditional linear MOS/EMOS methods.
• Learn dependencies across 173 weather features by post-processing them jointly rather than individually.
• Evaluate performance against baseline approaches and analyze suitable evaluation metrics for ML-based post-processing in meteorology.

Proposed method

• Post-process GFS forecasts (6h lead times to 7 days) to better match GDAS analyses at t+6.
• Predict the forecast error e_{t+6}=GFS_t^6 - GDAS_{t+6} using inputs including GFS_t^6, GDAS_t, sun position, and a decaying weighted bias term B_N.
• Project global fields to 0.25 x 0.25 degrees and tile into 256x256 grids to fit memory constraints.
• Compare a simple pointwise network (576K params) to a UNet-like model; select the simpler model as ML-PP due to comparable performance.
• Experiment with training schemes: single global model, latitude-weighted loss, and three-region (north, tropical, south) models; find three-region training best.
• Explore loss functions (MSE, MAE, Log Cosh, cosine similarity, Fractions Skill Score) and report no significant performance differences overall.

Figure 1 : RMSE of different weather variables at 500hPa and 850hPa.

Experimental results

Research questions

• RQ1 Can non-linear ML post-processing outperform traditional linear MOS/EMOS approaches for global, multi-variable NWP post-processing?
• RQ2 Does joint learning across 173 weather features capture inter-variable dependencies to improve forecasts?
• RQ3 What training strategy (global vs regional) and loss function yield the best global performance for ML-based post-processing?
• RQ4 How do ML post-processed forecasts compare to baselines in RMSE and Anomaly Correlation Coefficient (ACC) across leads up to 7 days?
• RQ5 What metrics best reflect forecast sharpness and spatial structure in ML post-processing?

Key findings

• ML post-processing yields RMSE improvements up to about 14% for wind components at 850 hPa and around 12% for temperature and precipitable water at various levels.
• Improvements of approximately 7.6% for geopotential height at 500 hPa and about 12–13% for other surface variables were observed in RMSE (7-day horizon).
• Three-region training (northern extratropics, tropical belt, southern extratropics) performed best, addressing regional differences in error characteristics.
• Statistical significance indicated some RMSE and ACC gains; geopotential height showed no ACC improvement from ML-PP, while other variables had modest ACC gains.
• The study highlights that conventional metrics like RMSE/ACC may not fully capture perceptual sharpness and leads to a call for alternative loss/metric design to preserve synoptic variability.

Figure 2 : Skill card with color-coding to indicate statistical significance for 7 day post-processing at different vertical levels. White blanks mean data at those levels was not available in GFS. Top: MSE for variables at multiple vertical levels, bottom-left: MSE for surface variables, bottom-rig